Replications are hard work that pays off

Professor Jörg Peters talks about his experience with Open Science

How did you arrive at Open Science?

Of course, I welcome the trend towards more empiricism and the study of causal effects – after all, we must test theories and the efficacy of policies. At the same time I realised very early in my own work that the methods celebrated as gold standard are robust regarding causality, but they have other weaknesses that are given little room in scholarly discourse. That is how I found my topic of external validity of RCTs and the replicability of causal studies in general. It raised my awareness of research transparency.

How do realise your own aspiration?

For one thing, we provide the data we have used for our publications afterwards and make them available to ensure replicability, like the majority of researchers now. Secondly, for several years now we have been pre-specifying the research questions for all studies ex ante in registers that are open to public access, for instance in the AEA RCT Registry, so that all our hypotheses are traceable. Implicitly we have been doing this before by documenting our research questions clearly in our project proposals. And thirdly, I believe reproducibility is important. The entire issue of research transparency only makes sense if findings are actually verified ex post. We need more replication and a wider discussion of results that doesn’t stop after publication.

What is your personal experience with replications?

I have been involved in three replications and more are in the pipeline. For all three we diagnosed difficulties regarding replicability. Two of these replications are re-analyses of the original data. In one case we now cooperate with the original authors on revising the original paper, in the other we have already published our commentary. In the third case we have replicated a prominent experiment from the US in a new experiment in Germany. The original authors showed that social comparison-based home energy reports are a cost-effective climate policy intervention in the US. We raise considerable doubt about their effectiveness outside the US. This doesn’t mean the original study has no value, but it raises questions whether the results can be generalised and thus whether they can be globally relevant for policy.

How do others react to these studies?

In my perception the research community reacts with interest. Almost all our colleagues welcome what we do. We have had very positive feedback from internationally influential researchers. You could possibly say: people become aware of us who wouldn’t have done so otherwise. But you certainly do not make friends all round. Some ask if such studies are detrimental to your own career or even to the credibility of science. And there is indeed no established culture of replication in economics. Another problem is that it is difficult to place a replication outside the original journals who thus act as gatekeepers for the critique of their own studies. I believe that replications and commentaries belong in the public debate where they can serve as the basis for further discussion. Not everybody must and will agree with the argument immediately.

Yet you get involved in this work nonetheless. What drives you?

I believe that empirical research can improve politics – of course only if it is done correctly and if it is reproducible. Therefore replications and Open Science are essential elements, especially in economic research which is often involved in important societal processes. If I have concrete doubts about a study I must question the impact, in public. I consider this a normal process of scholarly exchange. I suppose everyone would subscribe to this, but sadly it is barely implemented in practice.

You try to establish a culture of transparency in your research group. What is your approach?

We try to practise a culture of transparency regarding our own research, and we critically analyse and question the research of others. We don’t conceal difficulties in our studies but discuss them openly among us and document them for outsiders. But you have to be realistic: there is a great pressure to bang your own drum if you want to publish your paper in a good journal. I don’t pretend to be holier than the pope here. And we have learned a lot, I would write many previous papers differently today. But the key statements hold up, I think. In this sense, Open Science is more a process than a condition.

How important for your work is the exchange with other disciplines or institutions?

Exchange with other disciplines is very important because it enhances our work with other perspectives and opportunities. For instance, we collaborate with engineers on projects for building infrastructure in Ethiopia. Many questions that economists are working on have sociological or political dimensions that we should take into consideration more than we do.

But transdisciplinary collaboration can also move a lot. We are active in different networks that foster exhange with politics, as in the initiative Environment for Development. For many years we have been working on building infrastructure in the energy sector and on decentralised renewable solutions which gets lots of attention. We are in regular exchange with the World Bank where we have advised people at various levels, including the president. Our findings are acknowledged but of course they are not implemented exactly as suggested. But it would be strange if scientists were the only ones with influence on such complex political processes.  

The interview was conducted on November 19, 2020.

About Professor Jörg Peters

Jörg Peters heads the research group “Climate Change in Developing Countries” at the RWI – Leibniz-Institute for Economic Research in Essen and holds a chair at the University of Passau. His research interests are environmental, energy and development economics. He is also involved in the debate around research transparency and how to find the right evaluation methods for crucial political problems. Jörg Peters has advised various international organisations such as the World Bank, the Millennium Challenge Corporation (MCC) and the GIZ. His research findings have been published in leading journals, among them Journal of Health Economics, Nature Energy and World Bank Economic Review.

Contact: https://www.rwi-essen.de/peters

Twitter: https://twitter.com/jrgptrs

to Open Science Magazine