Open Science in economics
Thoughts by Professor Marianne Saam
Economists are probably aware of the replication or reproductions crisis. Even if they don’t experience it as a crisis in their own daily research, it has been gaining ground in the public eye over the last few years. Why exactly, and what changes can be observed? We have asked Professor Marianne Saam for her thoughts.
What role do the transparency and reproducibility of research findings play in economics?
MS: In a certain area of empirical methodology large progress has been made. Especially in microeconomics, the requirements for the identification of statistically causal effects have increased. Procedures such as instrumental variables methods and regression discontinuity designs are standard tools of empirical research nowadays. Reviewers at journals examine more critically than before whether causal methods are used and if they are justified plausiby. This year, Joshua Angrist, David Card and Guido Imbens were honoured for their pioneering contributions to the analysis of causal relationships with the Nobel Memorial Prize for Economics.
This sounds very welcome.
MS: Yes indeed. However, formal replication studies for verifying the validity and robustness of the findings in published papers in economics are still a minority. Informal replications are quite established as part of research practice, i.e. the findings of other authors are often emulated before they serve as the basis of one’s own work. This applies mostly to often-cited papers in high-ranking journals.
Is there a trend reversal in the offing?
MS: Yes, the American Economic Association (AEA) changed the requirements for its journals in 2019 and this could indeed herald a trend reversal. Data and programming codes must now be made available before an article is accepted for publication.
What happens if data are sensitive and cannot be made publicly accessible?
MS: If data are not publicly accessible, replicability must be proved otherwise, for instance wit a replication report from a neutral third party. The data editor ensures the implementation of the measures. It’s not the measure in itself that is new, but the consistency of practical implementation. The AEA journals thus comply with comparatively high Open Science standards. The ZBW offers the Journal Data Archive, free of cost, which enables editors to make the data and programme codes of articles available for reuse.
How is the publishing culture evolving with regard to replication studies?
MS: Economics as a discipline has no established practices for disseminating findings from so-called scholarly replications. Scholarly replications do not examine whether a study has been executed without technical faults, but verify instead the robustness of the results by using different methods or data. There are suggestions from the scholarly community that highly visibly journals could publish replications in the form of a detailed abstract. This could creative incentives for replications, by allowing a second publication as a separate part of an independently created article. This would give more visibility to replications that would have been done anyway. Citations should then refer not only to the original publication of a result, but also to validating and negative replications.
This makes sense.
MS: Yes. Independent journals for replication studies can also play a part, such as the Journal of Comments and Replications in Economics (JCRE) – successor of the International Journal for Reviews in Empirical Economics (IREE) – which is published by the ZBW. The journal could be used increasingly for the publication of replications in Empirical Economics that are created during PhD programmes.
What other changes do you observe with regard to Open Science?
MS: For experimental papers with random samples, the preregistration of a study is becoming standard. They state in advance which correlations are to be studied and therefore reduce biases caused by “p-hacking” and other approaches. A register set up by the AEA in 2013 now shows more than 4,000 entries per quarter.
Given the policy implications of economic research, society demands solid and consolidated results. How do economists discuss this?
MS: The political relevance of economic research is beyond dispute. The measures described up to here do not yet ensure that different findings to a question are consolidated. Methods of meta-analysis and of systematic literature review intend to create syntheses that are not dependent on purely subjective criteria. But they cannot remedy quality defects in the primary studies. They can provide additional information on questions that are widely discussed and where many researchers have been able to form an opinion at first hand about the quality of the previously published studies. The number of insights they could generate independently of an intensive discourse about individual studies seem to be limited, however. Which practice could offer the highest added value for economic policy is a question that economic research institutes, learned societies and science organisations could address more intensively in order to create standards for a more transparent policy consultation.
Thank you!
You can find the thoughts of Professor Marianne Saam on economics and Open Science in full length in Wirtschaftsdenst (in German). URL: http://hdl.handle.net/11108/500
The interview was conducted on December 8, 2021.
About Professor Marianne Saam
Marianne Saam is Professor of “Digital Economics” at Hamburg University and heads the programme division “Knowledge Transfer in Economics” of the ZBW – Leibniz Information Centre for Economic.
Her research focuses on the role of knowledge and digital technology in the economic process. She studies the effects of the digital transformation on innovation and sustainable economic growth, but also the use of digtal technologies in knowledge creation processes about economic topics.
Marianne Saam studied at Goethe University in Frankfurt on Main, where she also gained her PhD, and spent time in Paris and Louisiana. From 2005 to 2017, she headed various research projects at the department of “Digital Economy” at the ZEW – Leibniz Centre for European Economic Research in Mannheim. She has published papers in renowned journals such as the Review of Economics and Statistics. In 2014, she qualified as a professor at Goethe University. From 2017 until 2021, she held a chair at Ruhr University Bochum.
Contact: https://www.zbw.eu/de/marianne-saam
Twitter: https://twitter.com/mariannesaam
LinkedIn: https://www.linkedin.com/in/marianne-saam-432b95b9/
ORCID-ID: https://orcid.org/0000-0001-7055-0567