Critical questioning pays off for your career in the end

Dr Maren Duvendack talks about her experience with Open Science

You are very much involved in the Editorial Board of IREE – the replication journal for economic research. So I know that replications are close to your heart. Let’s get right down to it: how do your peers react to your work?

MD: Replication studies are close to my heart and I’ve been doing them for more than ten years. For my PhD thesis I looked at the study Micro-Finance in Bangladesh written by Pitt and Khandker and I found that I got different results from my reproduction. That’s when the drama started. The data of the study could be downloaded from the World Bank, but the datasets were incomplete and the code for retracing the analysis was not available, either. I contacted Pitt and Khandker, but they were uncooperative at first. After some time, and only with the help of David Roodman from the Center for Global Development who worked with the same data at the time, I could acquire the missing information. When the time came to publish the study, it proved difficult at first, because the editors considered a replication insufficient in itself. In the end the study was published but only because of an add-on. We applied a different econometric method to the dataset, carried out a check for robustness, so to speak. It caused resentment after publication, both from the original two authors and the microfinance lobby. That was very disagreeable, especially when you’re just starting your academic career. However, I quickly learned from conversations with my peers that this experience is neither surprising nor rare. And fortunately things have changed a lot in the last years since my own experience with replications, mainly in psychology and political science. Economists are still lagging behind, but things are looking up. The American Economic Review and other top economic journals now have data availability and replication guidelines authors must adhere to if they want to publish in these journals. Change is coming through the back door and is driven by the journals and other initiatives, such as the Replication Network and the founding of IREE.   

Which benefits of Open Science have you experienced? Do you have tips for other economists?

MD: As a result of my PhD thesis I received grants from the British government for a systematic literature review on microfinancing and its effectivity in Third World countries. Replications were an important feature here. And based on this follow-up study I wrote an impact case study. It’s part of the academic work in the UK to write down how your own research affects politics and society. As a result of my replication and the following systematic literature review, the view of microfinance in politics and the microfinance community has changed completely. It has become clear that microcredits are no longer the holy grail to reduce poverty and to empower women. No, it’s much more differentiated. So I have had an indirect influence on the British government regarding its microfinance investments in Third World countries. My studies have played a part in the political decision-making.

My critical approach has been super beneficial for my career, my network and my real-world impact. I have contributed to the establishment of better microcredit programmes and to an increase in the combining of different programmes. Replication and evaluation need not be negative, even if it is sometimes rocky at the beginning. You have to be courageous and stick to your critical position and its principles. At some point it will pay off. And now you can say: Duvendack started looking at microfinance critically. Now it’s mainstream.

What is the role of replications in policy consulting?

MD: In policy consulting it is always difficult if you focus on one study only. A single study always has limitations, especially regarding external validity, i.e. how can this study be applied to other situations, people and geographical context. Systematic literature reviews are optimal where you collect the evidence for a topic and often review thousands of studies. Such systematic literature reviews are currently much used by politicians in the UK, but increasingly also in other countries, for making decisions. Because they gather thousands of relevant studies on a certain topic and submit them to a quality check.

Has the way in which economics deals with replications changed since you wrote your PhD thesis?

MD: There has been progress, definitely. Five years ago Robert Reed and I founded the “Replication Network”. Many economists have joined who want to do replications, and the list of replication studies on our website is growing. Many economic journals now make it a pre-requisite for publishing that studies be replicable, i.e. data and code must be made available to the journal. Of course it’s more work to prepare data and code so they can be shared, but for me this is part of good scientific practice. And I as a researcher will be able to retrace five or ten years from now what I actually did there. So it’s actually quite helpful for oneself. 

Can you tell a bit more about the “Replication Network” you have founded? Who participates, what do you do?

MD: It’s an international network and we want to draw attention to replications and their importance. The focus is on economics because it didn’t have such a network. Robert Reed from Canterbury University (Christchurch, New Zealand) and I co-wrote a paper on which we based the “Replication Network”. It’s an informal network for economists worldwide, we have a pool of replication studies and a blog. We want to sensitise people to either doing replications of their own or to introducing students to the topic.

Where do your members come from?

MD: Most are from the Global North, often from English-speaking areas, so there are many from the US, UK, Europe, Australia. We also have more and more members from Asia, India, Malaysia, Taiwan; but it’s true that industrial nations dominate. It needs a bit more work to integrate economists from Africa.

Do you have members from Germany, Austria or Switzerland?

MD: Yes, we do have members from German-speaking countries, there’s been much more interest from Germany in the last few years. It’s nice to see the group interested in replication studies growing slowly.

What motivates economists to do replication studies?

MD: They are motivated by the conviction that replications are part of good scientific practice and useful to the research community. Especially for young scholars replications provide a wonderful means of understanding how data management and analysis work.

What are the benefits of your Replication Network?

MD: Scientific work can’t proceed without transparency. And replications are about transparency. I can increase my credibility with replication packets. I show that I work correctly and with scientific exactness. Practising replications myself allows me to give an understanding of the topic to others. And I can spread good scientific practice. In our Replication Network you get the latest news, the latest developments in the context of replications, an overview of different kinds of replication studies, and where you can publish your own etc. We have a blogspace where you can present your own studies and you can widen your network, so you can have exchanges with like-minded people and initiate further cooperations.

What benefits do you gain as the founder of the Replication Network?

MD: I get invited to conferences to talk about the topic, as keynote speaker or panelist. This widens the network into other academic areas that you normally don’t have much to do with. For instance, I was invited by a network of agroeconomists who do lots of experiments. All disciplines share the same problems, everyone lives in their academic silo and often there is very little exchange, alas. Our Replication Network leads to exchange between different academic disciplines. Exchange with psychologists is especially interesting because they are far ahead of economists in this area.

You have mentioned before that journals often demand replication packages. What is your view: has this become more during the last ten years?

MD: The trajectory has risen steeply, I would say. First of all, the number of replication studies has increased, but also the number of data availability guidelines, especially at the top economic journals. That is definitely a positive trend. But you have to differentiate between the policies and actual practice: journals often ask if you would provide your replication packet, but they do not always request it, because it is very time-consuming to verify everything. The American Economic Review is doing it and setting standards here. The AER for instance has data editors and PhD candidates who check the replication packets. This is what we should all end up with. But in reality science hasn’t achieved this yet. On the one hand, inclusion of the packets is wanted, on the other it eats up a lot of resources. Journals ought to hire extra data editors who are familiar with different software programmes (STATA, R, SIS) so they can verify everything. That is expensive, of course. In addition, the number of submissions has risen enormously due to the pandemic, so that verification of replication packets requires additional effort.

Do you use platforms for collaborative cooperation?

MD: I created a project I started a few months ago directly in Open Science Framework. I have done many systematic literature reviews where you also do a pre-registration in principle. So first comes the topic, then a pre-registration study in the form of a protocol, and based on this the actual full systematic review. The system of pre-registrations and pre-analysis plans is growing in the social sciences, in economics, too. The system has long been in place for systematic literature reviews. And since I have been doing systematic literature reviews for ten years, I am familiar with it. This principle is comparatively new in my evaluation research, but it is coming. I can see that.

When people do replications and are unable to falsify the original results, many desist from publication. What’s your opinion on this?

MD: It’s a serious problem. Many are worried when the results turn out differently than expected or if the original authors are unhappy, then people tend to self-censor. I have no other solution than to create awareness for this and to talk about it. Generally we can observe that many top economic journals increasingly establish their own replication sections or have special issues in replications, so that authors can publish their replication studies in a high-ranking journal.

You also teach. What is the role of Open Science and learning skepticism in this?

MD: I think it is very important to look for a critical discourse here. I try to encourage my students to question things critically and not just to accept them. We have students from all over the world and from very different education systems. I often need to explain what critical thinking is and how to question things. Sometimes I am shocked how uncritical students are. For their final theses I insist that students work transparently and provide codes and datasets. If the present generation of young researchers doesn’t practise Open Science, we as teachers have missed something important. Then it will become increasingly difficult to estblish openness as good scientific practice. What I also see is that our students work a lot with clean and well-organised secondary data packets, but they have no idea about how to manage and clean up raw data. This is not taught sufficiently in the field of economics here in the UK. Something needs to be done here. I only learned this myself in the course of my PhD thesis. And because I think it is fundamentally important, we have integrated it into our courses now. If I can’t manage my own data, how can I create a replication packet for someone else?

How far have replications been integrated into curricula?

MD: I can see that my master students and especially my PhD candidates are doing replications. But they worry very much that replications will harm their careers. There’s much aversion to potential conflict. They learn how to do it during their studies. In their life as researchers they no longer do it because they do not want to become inconveniently conspicuous. I see replications as an important step in scientific practice. You should work openly and transparently which is why I do not understand this resistance against replications. This is completely normal in the natural sciences, but still has to find its way into the social sciences, and it has to happen soon because the results have consequences for real life.

Are there places, besides initiatives, in mainstream economic research where meta-topics and methods are discussed?

MD: At the annual meeting of the American Economic Association replications are repeatedly on the agenda and there are also papers about it. A few years ago we were invited to Chicago to give a talk about it. So it is very much present as a topic.

How do you see the future of Open Science?

MD: My view is positive because ever more people are talking about Open Science and replications, in different disciplines and different journals. I think it depends on the academic field you are working in. In some disciplines the pace is quicker, among economists it is slower. But more and more people are working on it. I’m thinking here of the Berkeley Initiative for Transparency in the Social Sciences. The topic definitely won’t go away suddenly, because too many people have invested their lifeblood and want to take this forward. 

Last questions: what is your advice for young researchers?

MD: I would advise them to work with a key publication during their PhD thesis and to try to replicate it. You learn a lot through replication and maybe you encounter something that leads to new methods or concepts and thus to new innovative findings.

Thank you!

The interview was conducted on May 28, 2021.

About Dr Maren Duvendack

Maren Duvendack is Senior Lecturer at the School of International Development at University of East Anglia (UK), where she teaches mostly at the MSc Impact Evaluation for International Development research group. Her key research areas cover applied micro-econometrics, impact evaluation, systematic reviews and meta-analysis, microfinance, replication and reproduction of quantitative analyses as well as research ethics. After completing her PhD she worked as Postdoctoral Fellow at the International Food Policy Research Institute in Washington DC before joining the Overseas Development Institute in London as a Research Fellow in evaluation and impact assessment.

Contact: https://people.uea.ac.uk/m_duvendack

ORCID-ID: https://orcid.org/0000-0002-8125-9115

Twitter: https://twitter.com/marenduvendack

LinkedIn: https://www.linkedin.com/in/maren-duvendack-a332a5

to Open Science Magazine