Replication studies and their value for society
Why economists should do more replication studies
IREE (International Journal for Reviews in Empirical Economics) is a replication journal for empirical economics. A platform solely for the publication of replication studies. This raises the interesting question of why this kind of platform is deemed necessary. Does economics need this?
A neutral platform offers chances
Professor Martina Grunow, Managing Editor of the online journal, explains why the answer to this question must be a resounding “yes”. She says that such a platform is absolutely necessary for economists because otherwise their replication studies would have no chance of being noticed. Because other journals show little interest in publishing such studies.
To the podcast with Professor Martina Grunow (in German)
Usually, replication studies only stand a chance of being published if they falsify the results of the original study. The more widely known the original study is, the more lasting is the attention paid to the replication.
This is unfortunate, because replications are invaluable, especially if they turn out to be positive. It is a huge mistake to think the knowledge gained from such verification studies is negligible, or that they are a simple repetition of something done before.
Replication studies help scientists to discern which of the results generated by empirical studies are truly robust, generalisable and replicable. Results that are actually applicable for science in form of further research and follow-up studies, or for politics as a consultation basis.
Why do replication studies at all?
The purpose of replication studies is to discover which study results can provide a valid basis for political decision-making or continued research. In this respect replications are a service to society.
Once this is accepted, the perspective on replication studies changes automatically. This in turn offers the opportunity to realise the value of such scholarly work and the creative potential it holds. The challenge lies not simply in a precise repetition or verification of a completed study. Replication is first and foremost about a deep understanding of the original, so that the methodology or the analytic data can be fine-tuned in ways that bring new insights.
The question “Why was this done in this way originally?” must be answered in detail and in depth, before answers can be found for further questions such as “Why and how could this be done differently?” The new findings are fundamentally affected by what is changed from the original study and how it is changed. The replication thus must be at least as carefully designed as the original, if not better. Proceeding from their deep understanding of the original, researchers must be able to fine-tune their settings and to substantiate their actions.
The relevance for society
Researchers often bear great responsibility. If you consider how often findings from economic studies influence policy advice, the importance of replication studies becomes even more obvious. Politicians all too frequently base their decision-making on data that were never verified scientifically by replication. This is what replication studies provide.
These studies scrutinise the analytic process of the present data, because errors could have crept in that only come to light through replication. Only if the findings of the original study can be replicated a second time or even more often, do they turn into robust data material that can be reused by researchers and politicians. However, replication studies go beyond the mere duplication of results. The issue here is not merely the robustness of data, but also whether the study results can be generalised.
The applicability of findings
Can the results of the original study be reproduced only under the given conditions, or are they valid under different conditions, too? Is it possible to apply the conclusions drawn from analysing one sample to other, comparable groups? Knowing that study results can be applied to other settings or to enlarged groups can increase their value immensely. In the field of economics such questions are fundamentally important for evidence-based policy advice.
A replication study might evaluate if a study undertaken in Germany could be applied to other European countries. In such a case, the replication would use exactly the same methodology as in the original but it would be based on different data. This is the essence of a replication study. It verifies results by changing precisely one parameter. Either the methodology is changed while using the same data. Or the methodology is replicated exactly and the data to be analysed is changed.
Invaluable scholarly work
Replication scientists show their competence and creativity in the ways and methods of the modifications. The ingenuity shown in the fine-tuning of the settings indicates the quality of a replication study. And thus a scholarly work often condescendingly named a rerun becomes an exciting adventure in research at the highest level. This has nothing to do with copy-catting a scientific achievement.
Study findings can be numbered in millions. But which of these are truly robust? This question is immensely important for business and society, and it has been the subject of scientific analysis. A study in 2015 came to the dramatic conclusion that a mere 50 per cent of empirical results in the field of economics can be replicated. Which means in practice that you might just as well toss a coin.
There is a need to catch up
Replication studies could play their part in turning results of economic research into reliable guides for political decision-making. If there were enough of them, that is. The lack of such studies disregards that economics have societal relevance and political responsibility, for which replication studies could offer a useful scientific tool that could help filtering out robust data material.
The academic world has a lot of catching up to do in this respect. The replication crisis cannot be overcome as long as the publication count is the only yardstick for academic careers and competitive pressures don’t ease. One possible alternative would be to regard research as worthy in itself and to allocate corresponding time budgets.
Replication studies are a part of this research and thus must be included in the curricula. Science, politics and society would all benefit.
Academics are not the only ones who need to step up here, so do politicians. If only such data were admissible for advisory purposes that had been verified with a replication study, science would need to get moving. The burden of social responsibility is carried by many. It must not be shifted to researchers.
IREE – International Journal for Reviews in Empirical Economics