The problem of reproducibility in economics

Assistance for economic researchers: How to make your research reproducible

Illustration

In the scientific community, increasing attention is being drawn to the problem of reproducibility of research findings, especially in the discipline of economics. Several seminal studies have been tested under the microscope of reproducibility and quite a few have been found wanting (see Camerer, C.F., Dreber, A., Forsell, E. et al. (2016): Reed 2018; Ioannidis 2013; Miguel 2021; Chang 2017; and also Kasy 2021). Why does this happen and what exactly is reproducibility?

Replicability vs. reproducibility: a subtle distinction

Before we look at the details it’s important to clarify the difference between replicability and reproducibility. Both concepts are fundamental for an understanding of scientific reliability and are often used synonymously although they represent different aspects.

Reproducibility describes the ability to repeat a scientific study or experiment under the same conditions and to achieve the same results. Other researchers must be able to use the same data, methods and analyses to achieve similar results. Reproducibility is an important principle to ensure the validity and reliability of research outputs.

Replicability, on the other hand, refers to whether a study or experiment can be conducted by other researchers using the same approach and methodology to achieve similar results. In contrast to reproducibility, replicability doesn’t necessarily refers to the use of the original data and materials, but to whether the same research approach can be reproduced. Replicability often means the attempt to repeat a study or an experiment in another context, with different participants or at another time in order to verify the stability and general validity of the result.

So the difference is that reproducibility refers to the repetition of a study under the same conditions, whereas replicability means the repetition of the research design under similar, but not necessarily identical conditions. Both concepts are important to ensure the robustness and reliability of scientific findings.

Why replication matters so much to science

Replicability is an essential element of the scientific method. It serves to confirm the integrity of research findings and to ensure the impartiality of the original study. By repeating experiments and studies, scientists can uncover errors or bias in the original research. The pursuit of replicability thus contributes to the integrity and credibility of science.

The replication problem in economics

In the field of economics, studies are often based on complex models and large datasets with numerous variables. Access to these data, however, is frequently difficult because there are restrictions or the data are proprietary. This increases the difficulty of replicating or reproducing these studies.

A fundamental problem lies in the insufficient accessibility and also in possible errors during processing of the data and code. It can be proven mathematically that under certain assumptions the reproducibility of statistically significant results is affected by sample size and limited statistical test power. This becomes problematical if researchers tend to publish significant rather than insignificant results. This can exacerbate the challenges of reproducibility even further.

The problem sometimes goes beyond mere reproducibility and also regards the so-called post-study probability. This refers to how likely it is that a statistically significant effect is actually due to a true relationship. This brings other factors into play that affect the understanding and the interpretation of scientific findings.

Ensuring reproducibility and replicability in economic research is highly important. Only research findings that can be verified and replicated can be accepted as reliable and robust. It is therefore essential that access to data is made simpler and that transparent and reproducible methods are applied. Only then can we strengthen trust in the results of scientific studies and make well-founded decisions based on solid economic knowledge.

Assistance for economic researchers: How to make your research reproducible

Economists can use a number of strategies to make their research more reproducible and more transparent:

  • Data access: Researchers should try to make their data openly accessible as far as data protection rules permit. In this way, they open the path for other researchers to verify their results. Here’s a selection of suitable repositories:
    • Zenodo: a general Open Access Repository developed and supported by CERN, the European Organisation for Nuclear Research. It enables researchers from all disciplines to share their data.
    • Registry of Research Data Repositories (re3data): a global index of research data repositories which helps researchers find a suitable repository for their data.
    • Datorium: a repository provided by GESIS since 2014 for researchers from the social sciences and economics. Imported data are assigned a DOI and are persistently held available in the form originally submitted.
    • Research Data Centre for Business and Organisational Data (RDC-BO): a central archive for quantitative and qualitative data from business and organisational research at DIW Berlin.
    • There are many universities in Germany now which offer storage for research data on their own internal publication servers. We suggest you look for recent information at your university or research institution.
  • Methodological transparency: Researchers should describe their methods extensively and transparently. This makes it easier for others to understand and to repeat the research process.
  • Standardisation: By using standardised methods and analysis tools, researchers can ensure that their results can be reproduced easily.
  • Pre-registration of studies: Pre-registration of their studies allows scientists to define their hypotheses and methods before they start their studies which increases the credibility of their findings. There are several platforms where economists can pre-register their studies:

It’s important to keep in mind that pre-registration is not simply a formal requirement. It is a strategic decision which induces researchers to plan their studies more thoroughly and make them more transparent.

Replicability is not merely a technical requirement, but a principle of scientific integrity. By improving reproducibility, economists can help increase confidence in their work and improve the overall quality of research.

Exercise: “The seat of a replicator”

Finally, an exercise from Charlotte R. Pennington’s book: A Student’s Guide to Open Science 2023, p. 21):

Find your favourite research article from the journal where it was published. See whether there is enough information reported for you to conduct an independent replication.

Ask yourself the following questions:

  1. What are the aims or goals of the study?
  2. Are there any hypotheses? Are they clearly stated?
  3. What is the research design?
  4. Who are the sample participants?
  5. What is the sample size? Do they provide a rationale for this?
  6. What is the independent variable (the manipulated variable)?
  7. What is the dependent variable (the measured variable)?
  8. What analyses do they conduct? Is there enough information for you to reproduce these?
  9. What is the threshold for reporting statistical significance, or evidence for their hypotheses? Do they provide the alpha level (p-value) used?
  10. Do they report whether they screened or cleaned their data? (e.g. Were their inclusion/exclusion criteria? What did they do with outliers in their data?)
  11. What information is missing from the study which might make a replication challenging?
  12. Do you expect this study to replicate? Why or why not?




to Open Science Magazine