It must become easier to publish insignificant results because they show courage

Professor Urs Fischbacher talks about his Open Science vision

You work in experimental and behavourial economics. Do pre-registrations matter to you? 

UF: Pre-registrations are finding entry into the research process. They are especially useful where things are difficult to replicate. I don’t think pre-registrations are important for easily reproducible studies, for instance for normal lab experiments, because these lab experiments can be easily reproduced if you have doubts about something. Also, truly important studies are replicated automatically when new studies build on other studies. The problem as I see it is that replications sometimes fail. And then the whole project fails, and the information about the failure remains locked in the drawer. It is plainly absurd that unreplicable results are replicated so often because knowledge about them is not disseminated.

Are there better solutions?

UF: The journal Experimental Economics gave us the opportunity to review a few articles in a pre-result review procedure. In this, an article is reviewed before the data are collected. I see such pre-result reviews as a better option than pre-registrations. With these you not only prove that you already had your hypothesis. The paper can be published regardless of the outcome, because the reviewer primarily states whether the question in itself is interesting and the author knows the paper is accepted for publication before collecting data. Once the data are collected it is of course possible to do explorative analyses, which can be criticised again by the reviewers. But the core of the article is safe. I think this pre-result review is a promising approach for risky experiments that are nonetheless interesting.

You looked at honesty in your die experiment. Can you draw conclusions from this experiment on the importance of transparency?

UF: When there is transparency, it is impossible to be dishonest. If I as a researcher have to lay open my instructions and my programme I cannot claim to have carried out my experiment differently. When there is pre-registration, I cannot change my hypothesis afterwards. What I find exciting about honesty is to see if people believe the others are cheating, if there is a social norm for honesty. And in this respect a lot has changed for the better. People read much more skeptically than they did years ago, especially regarding ex ante hypotheses. Of course, that creates an incentive, e.g. for pre-registrations. In this sense transparency helps improve the quality and reproducibility of results. In my view the problem is not so much that unsafe data are published, but that insignificant results remain unpublished and you thus arrive at a wrong assessment of the relevance of an effect.

Have you yourself published insignificant results?

UF: Yes, indeed I have. But it was relatively difficult. If I’m working on two projects and one of them has beautiful results, then working with the interesting results promises more success. You get into the top journals with spectacular findings, not with spectacular designs. And that’s a huge problem. That’s also why for me pre-result review is more interesting than pre-registration, because with a pre-result review publication is guaranteed and it requires less effort.

Do you publish your method, codes, algorithms etc.?

UF: When I submit a paper, the instructions are included in the paper or working paper. Data and code are usually deposited with the journal for publication. I think it is a good practice because in this way authors are forced to share their research completely. If there are no legal impediments, I usually provide a replication packet. It’s not always convenient, because programmed experiments have a certain complexity. It requires effort to provide documentation in a way that allows others to reproduce everything. I also think that publishing replication packets through journals or data repositories is much more sustainable than doing it on your own website. It has happened to me, too, that I joined another university and my profile page, where the annex to a paper was linked, vanished. In my opinion, journals should be responsible for making these data available sustainably. The government must provide solutions and infrastructure to ensure that access to these data is free for all.

You have developed the programming language z-Tree which has been available freely as cite ware since 1998. Which benefits have you derived from sharing it openly?

UF: The programming language z-Tree is in use worldwide and has definitely helped me increase my international visibility. You can track its use with Google Scholar searches and there are more than 10,000 papers that have applied z-Tree. In 2016, I received the economy award of the Joachim Herz Foundation, and one of the reasons was that the global dissemination of z-Tree played a significant role in making this branch of research internationally important. The programme makes it much easier to conduct experiments and sharing is also much easier if many people are active on the same platform.

Wo Where do you as a behavourial economist see further incentives for Open Science?

UF: I see two things: research data must be made available before the article is published. Journals must carry out the quality checks for these research data and test for minimum requirements such as whether the data scripts run or not. The second thing, which I consider to be extremely important, is that insignificant results should be easier to publish. We have already considered a Journal of Null results which would be extremely important for meta-analyses. We only get to see censored data, we don’t get to see the insignificant ones. This should change. The corresponding articles could be simple and short, but the authors should earn recognition for them, because null results are a sign of courage.

Which contexts do you see where economic research discusses Open Science and practices of integrity?

UF: The topic manifests mostly in practice. There are more pre-registrations. Journals are more willing to publish insignificant results than in earlier years – but still too few. Experimental Economics has a sister journal, Journal of the Economic Science Association (JESA), which explicitly invites submission of replications. The replication study of Open Science Collaboration is discussed among economists and a similar study has been carried out in our discipline, too. It is an important aspect that all economic researchers are aware of the problems.

Thank you!

The interview was conducted on April 6, 2021.

About Professor Urs Fischbacher

Urs Fischbacher is professor of Applied Economics at the University of Konstanz. He heads the Thurgau Institute Of Economics which is affiliated to the University of Konstanz. His research activities cover experimental, behavourial and neuroeconomics. He is known for developing z-Tree, a programming language for conducting scientific laboratory experiments which is used in many research institutions and which has played an essential role in establishing the importance of this branch of research. In 2016, he received the Joachim Herz Economy Award. In the German F.A.Z. Ökonomenranking 2020, Urs Fischbacher was ranked in fourth place of top economists in the category „Science“.      

Kontakt: https://www.wiwi.uni-konstanz.de/fischbacher/members/urs-fischbacher/

ORCID-ID: https://orcid.org/0000-0002-5115-8815

to Open Science Magazine