Pre-registrations are the path to the future

Andreas Dür talks about his experience with Open Science

Which Open Science practices have given you an advantage in your career?

AD: You try to apply Open Science practices in every project. To me it was important that we published the dataset for the project “The Design of Preferential Trade Agreements” early on. This dataset has been used by many people, either by economists or political scientists. It’s become a kind of unique selling point for the research I do myself. And it’s just an example because we have been very open about our data. It was cited a lot, and because we were so open we received a lot of feedback that we could integrate into new versions of the dataset. So we’re keeping the dataset up-to-date and we continuously correct mistakes.

You are a member of the Editorial Board of the Institute for Replication. According to the website, part of your work is to “actively recruit and select replicators”. How do you find people who replicate?

AD: The team of the Chair of the Institute for Replication, Abel Brideur, compiles lists of all articles published in the top journals of economics and political science. In a first step, these people check if there’s enough replicable material available, otherwise it won’t go any further. If there’s no material, they contact the author(s) and ask them to make the material available. That gives a list of studies which should be replicated. And we, the Editorial Board, review this list and look at studies from our own field of research. Then we pass on names that the Institute for Replication should approach centrally from Canada. Quite often we get a positive answer.

Who reviews the replication?

AD: Abel Brodeur and his co-workers review the replication study and check if everything fits. Then it is passed on to the authors of the original study with a request for comment.

Is there a mentoring system in place for early career researchers who do replications?

AD: That’s the next step that could come. Right now there are Replication Games, where many people in groups try to replicate various studies in a single day – like a hackathon in computer science, it’s a similar idea. There we find professors who replicate a study with five students. That would really be like showing how to do it in real life “step-by-step”. Yesterday the Oslo Replication Games were held where people from all over the world came together. Quite a successful format, actually.

How important are replications in your teaching or PhD education?

AD: When we teach research design to students or PhD candidates, one of the central issues is that everything must be transparent and reproducible. Of course, in an ideal world you would have several weeks and try to replicate a study. That would be ideal training for students, in my opinion.

Does the Institute for Replications have a political agenda? For example by suggesting that for a cumulative PhD two replication studies should be included? Or to make them a mandatory part of PhD training?

AD: I haven’t heard that yet. But it’s an exciting idea. You have to find a balance between the replicating part of a PhD or master thesis, and the part that creates something new. I think it’s exciting. But you would have to think carefully what you can demand and which exemptions there should be.

Some researchers are total fans of pre-registrations. Their pro argument is that it strengthens the methodological section, reduces publication bias and is partly linked to a publication guarantee for young researchers. On the other hand there are doubters who feel their flexibility is curbed and say that pre-registrations are to the disadvantage of explorative elements. What is your experience with pre-registrations?

AD: First of all, I believe that pre-registrations are the path to the future. We have to pre-register. Replicating something is okay, but we don’t know how much p-hacking has taken place if it hasn’t been pre-registered. I can replicate the p-hacking and it will produce the same result. I can’t solve the problem like this. That’s why I believe we must pre-register our studies. Of course there are several factors coming into play. How exactly does that look like? How many hypotheses do you formulate? I’ve seen pre-registrations with forty hypotheses. That’s no use, it’s just poking in the mist.

You mentioned flexibility. How do you deal with it when, for example, the results do not correspond to the hypotheses? Do you dismiss everything? But then it implicitly becomes p-hacking, because the studies vanish which don’t show it. And those who accidentally show it are published. That’s also a kind of p-hacking, just another kind, across many studies and researchers. That’s also difficult. Then there’s the question of timing. When do I need to submit my pre-registration? If I have a trial dataset in front of me with 1,000 responses I can already see a lot. Is before or after the trial a good time? There aren’t any firm rules yet. We’re all still experimenting to find out what’s ideal. And the publication guarantee you mentioned is very rare, at least in political sciences. There’s a pre-registration and then you look for a journal. Of course it would be super for meta-analyses if insignificant results would also get published after pre-registration.

Is pre-registration a standard operating procedure for you when you work empirically?

AD: I have now made a habit of it for experimental studies. It has become standard for me. It’s still different when I work with observation data, and I’m not quite sure how that would work. It can be difficult to do that in some experimental studies, too. For example, I have run an experiment with members of parliaments, and I don’t know in advance what the response rate will be like. Every step was a trial, and if you’re the first to try something like this, you don’t know beforehand what the path will be and then you can’t pre-register it properly. So there are various areas where I have found it difficult myself to deal with it.

What are the benefits of pre-registration for different career stages, in your opinion?

AD: I think it does not depend on the stage of your career. I have been in this business for quite a few years and sometimes you make decisions too quickly, go into the field and afterwards you’re wiser. Recently I ran an experiment with a PhD candidate and a PostDoc which we pre-registered. The data will arrive in a few days. The interesting thing about it was that we presented the pre-registration at conferences. Before we went into the field we already obtained comments from other people, discussed it, and that has been extremely useful. We’ve had a lot of input for this. That’s something I really want to carry over into other projects.

Do you pre-register with Open Science Framework? Do you recommend it to your PhD candidates?

AD: I am now using Open Science Framework, and I also recommend it to PhD candidates. For them, an OSF template is a big help. They can check if they have thought of everything before they go into the field. It’s enormously helpful to be clear about the research design before you start with a study.

How important are pre-registrations in your discipline?

AD: In the domain of experimental research it’s becoming standard. Three, four years ago you would have run an experiment and not even thought about it. In 2022, it’s completely normal.

How did this get accepted so quickly?

AD: That’s a good question. The problem of non-replicability has been discussed for quite some time. At some point it became clear that there is only one solution for the problem. And that is to pre-register the studies in advance and to state the various operationalisation steps clearly in advance as far as possible. So that we researchers don’t have that many degrees of freedom afterwards. Of course there were a few journals who were riding at the front of the wave and said we don’t accept any more studies that aren’t pre-registered. And as soon as a few important journals in the discipline do it, it will be widely discussed, at least. And after that, there’s no escape any more.

Do the journals lay down a set of rules or are there actors in your field who say how things must be done?

AD: As far as I know there’s no good set of rules yet in this area. And the journals are also still pretty vague about it. Maybe you shouldn’t set down rules so early on, but allow for a time of trial so we can see how does this work, what works best etc. Then you can lay down rules.

Open Science comprises a lot of different practices, from publishing in Open Access, publishing data and codes, pre-registration and even science communication with non-academic stakeholders. Where do you see the present state of development?

AD: We’ll probably never reach a final state. I think once we’re done with pre-registration we will have reached a mature stage. Until now I’ve always seen gaps, because p-hacking was always possible. And I still see one big problem: if I want to work with data from different statistical offices I encounter big gaps. Some statistical offices give absolutely no access to data, others only to scientists resident in their own state, as in Sweden, for example. Or I can only get access under strict conditions. Austrian data must be deleted afterwards, for example. And in Germany I must pay for the data. If people want to replicate the results of such studies they must pay just to get hold of the data. So it’s most unlikely that someone will take the trouble to replicate.

This is interesting. Why can data not be made available for research, in your opinion?

AD: You have talked to Harald Oberhofer, who founded the “Platform Research with Registers” with a few others explicitly to get better general access to data from Austrian administrations. And there is now a Austrian Micro Data Centre intended to give researchers access to public data. I pointed it out to one of my PhD candidates. But the restrictions were endless and the options very much limited. Obviously there are players who slow this down a lot.

Is there an actor in this whole science cosmos who is mainly responsible for the situation?

AD: It’s an interesting fact that it’s not the journals and scholarly societies who drive this issue, but influential individuals who address and promote the idea. Others pick up on it and then the wave starts rolling. First some journals get moving and finally the scholarly societies where many still want to block the movement.

How do you see the future development of Open Science? Will pre-registration become standard one day?

AD: This will never work without some pressure. I believe a certain pressure from the journals is needed, they must really demand it. The scholarly societies must implement it. People like me who train PhD candidates must pass it on that this is to be the standard. And I think in the medium term it will completely prevail among those who aspire to contribute to the top debates internationally. If you don’t provide the data etc, you won’t be taken seriously. I also see a great need for action in the field of qualitative research in political sciences, where I meet with problems of a different kind. How do I anonymise interviews if I talk about concrete issues that, maybe, only one person knows about. But qualitative research is already working on this, on how to present this more transparently.

What would you recommend to young researchers, besides transparency?

AD: My recommendation is to write down every decision from the very beginning of a project, to produce scripts that are replicable and that you yourself can still read a few years from now. And to generate data in such ways that they can be published. In brief, to use and implement this whole process of Open Science from the very beginning. If you do this from the word go, if you’ve got into the habit of doing things this way, then it’s also the easiest way.

Thank you!

Dr Doreen Siegfried conducted this interview on 28 October 2022.

About Professor Andreas Dür

Andreas Dür is Professor of International Politics at the Department of Political Science and Sociology at the University of Salzburg, Austria. He holds a PhD from the European University Institute in Florence (2004). Prior to taking up his current position, he was a research fellow at the Mannheim Centre for European Social Research and a lecturer at University College Dublin.

Professor Andreas Dür is a member of the Editorial Board of the Institute for Replication. The Institute for Replication (I4R) works to improve the credibility of science by systematically reproducing and replicating research findings in leading academic journals.

Contact: https://sites.google.com/site/andduer

ORCID-ID: https://orcid.org/0000-0001-6621-7243

Twitter: https://twitter.com/andduer

LinkedIn: https://www.linkedin.com/in/andreas-d%C3%BCr-0909298b/

ResearchGate: https://www.researchgate.net/profile/Andreas-Duer

to Open Science Magazine