Replication as the key to transparency
Interview with Chuan Liu about the role of the Journal of Comments and Replications in Economics
Foto: Maxim Schulz
There are nowadays journals that publish replication studies. What is the idea behind the Journal of Comments and Replications in Economics (JCRE)? Why such an outlet?
CL: While some journals offer to publish replication studies, such as the Journal of Applied Econometrics or several journals that edit special issues, these works often compete for limited space with original research. Also, many journals are reluctant to publish replications. So, JCRE provides a place for replication and comment works to promote constructive scientific dialogue among researchers. Whether the papers confirm or fail to replicate the original studies does not come into our consideration when assessing the submission.
What advantages does JCRE offer authors who submit their replication studies to you compared to other journals?
CL: In JCRE, our editors are more familiar with replication work, and we are trying to achieve a shorter turnover time. Besides, we require the submission of replication packages (that is: data and program code used) where possible. We check these packages for reproducibility in an in-house data review.
Who is providing the resources for JCRE?
CL: JCRE is currently funded jointly by ZBW and the Joachim Herz Foundation. Previous funding that helped develop and implement the journal idea came from the German Science Foundation (DFG). As the managing editor, I perform the operative tasks of the journal together with a student assistant and with colleagues at ZBW in charge of the infrastructure resources (the submission system, the website and the Journal Data Archive). Responsible for selecting articles for JCRE are Robert W. Reed, (editor, University of Canterbury, New Zealand), Dennis Wesselbaum (co-editor, University of Otago, New Zealand) and Marianne Saam (co-editor, ZBW).
How do you ensure the quality of the published comments and replications, especially when it comes to the reproducibility of the results?
CL: First, we implement the standard procedures of a peer-reviewed journal. This includes editorial screening, peer review and editorial decisions that take the peer review into account. If the evaluation in this process is positive, a student assistant will check the replication package to ensure that the replication package is push-button reproducible. This feature usually involves careful design of the directory setting and master script to run other code scripts. In most cases, the editorial team will use the student assistant’s suggestions to provide a data check report and help the authors improve their replication package. If data are inaccessible to JCRE for plausible reasons, a code check will be performed without achieving push-button reproducibility.
What criteria are used to assess the quality of replication studies?
CL: The journal will publish reproductions, replications using different methods or data, and commentaries on the original published work. The original study should be published in a recognised scientific journal. For replications, we require a standard format that includes a brief overview of the original study and its significance, a replication of the results of the original study, with the results of the original study and the corresponding estimates of the replication study presented side by side in a table, a presentation of robustness checks (if the replication includes any), and a summary of findings that were confirmed or not confirmed. For commentaries, authors must document that the submission was rejected by the journal that published the original study. In this way, we aim to encourage scientific exchange in the original journal before considering publication in JCRE. Beyond the guidelines provided on the website, the criteria are only those of standard peer review. Peer reviewers comment on the academic quality of the paper, and the editor or co-editors take into account reviews and revisions in response to reviews when making their decision.
A unique feature that distinguishes JCRE’s policy from most traditional journals is that we also publish pure reproductions that merely confirm the original analysis. JCRE believes that such publications provide valuable information about the validity of research. As long as the original research has made a significant contribution to the literature and the reproduction is presented in a transparent manner, JCRE welcomes such submissions.
How is the commitment to open science ensured at JCRE, especially with regard to access to data and reproducibility of studies?
CL: We cooperate with a ZBW data infrastructure, the Journal Data Archive, to require authors to upload their replication packages there and document the codes and data with sufficient information. Each replication package will have one Digital Object Identifier (DOI), a permanent code to be identified online. All replication packages are released as open-source materials under the Creative Commons License—Attribution 4.0 International (CC BY 4.0).
How do you promote scientific discussion between the authors of the original studies and the authors of the replication studies?
CL: In the review process, we usually invite one original author to serve as a referee. If they agree, we will inform the author of the replication study about this arrangement. So, in this case, this is a single-blind review. After the publication, we provide the original study’s authors with an opportunity to respond and publish their replies alongside the replication study on our publication page. We believe this kind of communication will benefit individual work and literature threads and create an atmosphere of openness and integrity in research.
What measures does JCRE take to ensure that all submissions are written in a civil and scientific tone?
CL: We have recently added a section on non-judgmental language to our submission guidelines. Authors acknowledge the inherent subjectivity of scientific analysis and contribute to a more open and respectful scientific dialogue.”
Let me explain further. The replicator might not be able to reproduce the results of the original paper. To some extent, disagreement could lead to uncomfortable feelings on both sides of this exercise. However, we try to promote the idea that replication is not about attacking specific authors. If a study cannot be fully replicated, this may be due to unintentional coding errors, which most researchers make at some point in their work. Such factual errors should be pointed out without blaming the authors. A (hypothetical) example of such an error might be that the original paper describes a dummy variable that assigns a value of 0 to small firms and 1 to large firms, while the code assigns a value of 0 to large firms. Especially when the disagreement goes beyond such factual errors, authors of replications should point out the subjectivity of their assessment and leave it to the reader to judge the validity of both sides of the argument.
What steps are you taking to increase awareness and acceptance of replication studies in the economics community?
CL: We are happy to have the support of researchers on our advisory board who are very prominent empirical economists and pioneers in the field of replication. We are also contacting PhD supervisors in top economics departments and asking them to encourage their students to submit their replication work to our journal. We are also planning special issues with a particular focus. We are part of what I would call an emerging replication ecosystem in economics, which is currently very dynamic, with initiatives such as the Institute for Replications led by Abel Brodeur at the University of Ottawa. We are optimistic that this ecosystem as a whole is in the process of increasing awareness and acceptance.
What types of analyses or research findings do you see most often in submissions to JCRE?
CL: Besides some replication works confirming the reproducibility of the original studies, many find that the choice of empirical specifications and variables can lead to quite different results or make the originally significant results less significant. Some original studies are important seminal works or recent publications with important policy implications. Such new findings published in our journal will encourage other researchers to investigate such topics more deeply and contribute to the robustness and credibility of economic research.
How has the submission and publication of replication studies developed since JCRE was founded?
CL: As a relatively new journal, we are gaining momentum and attracting more submissions. Seven papers have been published in our journal so far this year. Six of them are replication studies, and one is a comment paper. We hope to have stable 15-20 publications two or three years from now.
What role do you see for artificial intelligence (AI) in supporting replication studies, particularly with regard to the automation of data analyses and checking the reproducibility of research results?
CL: Looking ahead, we see potential in using artificial intelligence to enhance the efficiency of checking reproducibility. AI could also inspire new ideas in scientific replication, such as experimenting with different empirical settings or using new datasets. Last year, the Institute for Replication organized an AI replication game in partnership with the Labor Dynamics Institute at Cornell University.
How do you plan to further develop the JCRE in the coming years and strengthen its role in economic research?
CL: Attracting more submissions, for example by the special issues I have mentioned, and deepening contacts within the replication community are our most important steps for the next years.
Thank you very much for the interview!
The interview was conducted on October 18, 2024 by Dr. Doreen Siegfried.
About Chuan Liu:
Chuan Liu is Managing Editor of the Journal of Comments and Replication in Economics (JCRE). He is also a doctoral candidate at the Chair of Digital Economics at Prof. Dr. Marianne Saam since September 2021. His research focuses on economic growth, digitalization, global value chains, and static and dynamic input-output analysis.