Lab²: Promoting efficiency and transparency in research

You are the Co-Principal Investigator of Lab², a hub for replications and meta-science, supported by the Leibniz Association. Could you start by explaining what the main task of Lab2 is and how it came about?

LN: With the most abstract definition, Lab² is a replication and meta-science incubator. With a large network, we organize, participate in, and support the research community to engage in a large variety of studies. Those include one-to-one replications, conceptual replications, meta-analyses, many-designs, many-labs, and many-analysts studies. We provide training materials on replications, open-science, and transparency. Furthermore, we organize events, and give technical, intellectual, and financial assistance to junior researchers.

What specific inefficiencies in research does Lab² address and what strategies are used to minimize them?

LN: One of the major challenges in experimental economic research is the fact that laboratory experiments suffer from power problems (i.e. small sample sizes) and external validity problems. Given that researchers are under strong pressure to publish, and that significant results are obviously better published, they try hard to find significant results in their datasets. We refer to these efforts as “research degrees of freedom”, where researchers are free to choose the analysis path that leads them to a significant result.

Moreover, unlike the natural sciences, social science experiments are influenced by many unobservable factors. Unlike a biology lab, participants in economics come to the lab with their socio-economic history, their emotions, their daily moods, their political concerns, and many other factors that influence their behaviour. While student samples are more homogeneous and comparable, social science experiments are inevitably less replicable than natural science experiments.

In Lab², we strongly encourage the use of pre-analysis plans (PAPs). A PAP is a detailed version of a pre-registration. It contains precise and detailed information on how the data will be collected, how the dataset will be generated, analysed and interpreted. It should be prepared and submitted to an online repository prior to data collection. So when the dataset is ready, the analysis is ready.

In addition to PAPs, we also support collaborative studies, such as many-labs studies. In a many-lab study, several labs run the same experiment using the same instructions and code. Data analysis is also performed in a standardised way. Such studies provide strong and reliable results, rather than a collection of small studies with similar designs but inconclusive results. These types of large collaborations can therefore help to reduce inefficiencies in research.

How does Lab² contribute to reducing inequalities in access to information? What contribution does Lab make to Open Science?

LN: Lab² supports transparency and open science in all phases of research. This includes even the idea development phase. We will run a series of “behind the scenes” seminars and document them in various media formats to show that academic success comes with many rejections and mistakes. Neither social media nor researchers tend to show the full depth of the iceberg.

The Open Science Framework (OSF) and the ZBW are collaboration partners of Lab², with whom we will work to strengthen scientific norms of best practice. We also have a visual artist on board who will help us create and disseminate training materials such as posters, videos and a mobile app.

How is collaboration with different institutions organised within the Lab² core research team?

LN: There are several types of collaboration within the organisation. First, many of us have been co-authors or co-authors of co-authors. So we are aware of each other’s research and are in direct or indirect contact. However, the research agenda of Lab² should not be a niche topic carried out by the same network of researchers. Therefore, we want to reach out to researchers who have not been involved in replication or meta-science studies. That is why we publish our studies. Fortunately, we receive many emails from researchers around the world expressing interest in collaborating with us.

But I would also like to emphasise that we do not intend to run Lab² in a centralised way. In other words, we encourage both Lab2 members and the research community to organise their own crowd science projects, replication studies or events. We are always ready to support them with our connections, experience and resources.

According to your latest surveys, how widespread is the use of pre-registration and pre-analysis plans in experimental economics?

LN: The general picture is quite promising. Especially for pre-registration. We did a detailed survey of about 500 participants at the Economic Science Association World Meeting in 2023. 86% of them said they had pre-registered at least one of their studies so far. Further responses show that the experimental economics community is very sympathetic to pre-registration. But they also have concerns. For example, they want to be able to carry out exploratory analyses even if they have not pre-registered. They are aware that they need to be transparent about how strictly they follow their plan and what is not pre-registered.

However, there is great heterogeneity in the content of pre-registration and pre-analysis plans. Some researchers produce very detailed documents, while others produce very vague plans. As most journals do not require or provide guidelines for pre-registrations and pre-analysis plans, simplistic pre-registrations are naturally more common.

It is also not entirely clear how PAPs work. For example, do reviewers and editors rate papers with PAPs more positively? Do researchers who write detailed PAPs also write better papers, and what is the causality? Such questions are difficult to answer with the data available, but we will learn more about the mechanisms as the project progresses.

Can you talk about the ongoing efforts to understand the importance of different research choices through multi-design or analytic studies?

LN: There are three different types of heterogeneity that our research agenda aims to address: Population, design and analysis heterogeneity. Many-lab, many-design and many-analyst studies correspond to each of these.

In many-lab studies, we can look at how results vary from lab to lab. We can extend our many-lab studies to other populations, such as online participants, the general population, representative samples, or special samples such as professionals or affluent samples. We are currently developing a many-lab study on the joint role of material, cultural, cognitive and social drivers of collaboration. We aim to organise many more such studies in the coming years.

Many-designs studies are a new method that we are excited about. The first many-designs study organised or conducted by Lab² researchers was on competition and moral behaviour. 45 experiments investigating a single research question were run in Prolific with different designs. These 45 experiments were then included in a meta-analysis. This study was published in PNAS last year and involved more than 80 co-authors. A new many designs study on carbon pricing is being organised by our colleagues at the University of Innsbruck. Finally, we are finalising our preparations for a many-analyst study based on SOEP data. We are in close contact with our colleagues at DIW.

How does Lab² help researchers tell the stories behind their projects through surveys or interviews?

LN: Through the behind-the-scenes seminars and the use of different media formats, we aim to disseminate success and failure stories under the iceberg of academic success. We think this is very important because young researchers are particularly vulnerable to fierce competition and uncertainty in their professional and personal lives. Exposure to the stories of their peers and seniors can give them a more realistic and transparent view of scientific careers.

Are there any specific projects or results from Lab2 that you would consider particularly innovative or influential?

LN: Lab² is a new initiative and will officially start in October 2024, but its members have published important studies in the past.

Anna Dreber and Magnus Johannesson have been working on reproducibility and meta-science for a long time. “Estimating the reproducibility of psychological science” (Science, 2015) and “Evaluating the replicability of laboratory experiments in economics” (Science, 2016) are two of the most influential large-scale studies published in the last decade.

A recent study, “Nonstandard errors” (Journal of Finance, 2024), is also a major methodological contribution to the field.

Our recent paper “Does cognitive reflection relate to economic preferences and socio-economic outcomes” (Journal of Political Economy Microeconomics, Forthcoming) is a study I really enjoyed working on. It is a good example of how to investigate the external validity of laboratory findings in household panels.

What is the typical life cycle of a research project at Lab2, from idea to publication?

LN: It depends on the study and the methodology. If Lab² is organising the study, we might have a few meetings. But most of the time we work interactively on a document in the cloud, and we do it quite quickly. This document becomes our PAP. Then we register the PAP and continue with the data collection / dataset creation. Once the data collection is done, analysing the data and writing the paper is also quite fast. Because we have the PAP to guide us.

What specific challenges do you face when integrating new technologies or methods into the Lab² platform?

LN: A major challenge is the differences between legal requirements and practices in different laboratories. In some countries, researchers cannot easily pay participants. Some labs pay participants in cash, others make bank transfers. Some labs do not have funds to participate in trials. In these cases, it is quite difficult to send or receive funds. German accounting rules make it particularly difficult to transfer resources abroad, although international collaboration is supposedly encouraged.

There are also some differences in the software used. For example, some labs use ORSEE, a common participant recruitment platform, while others use SONA. Statistical tools can also vary. For example, in our many analysts study, we have to use STATA rather than other software such as Matlab to facilitate large scale collaboration.

What does the future hold for Lab²? Are there any new projects or expansions planned?

LN: We want to run and expand Lab² continuously. Therefore, rather than rushing into too many studies and activities, we want to grow carefully and systematically. The first year of the project is particularly important.

While we have many projects and ideas to develop, we are careful about two things: First, we should have enough resources (e.g. time, budget, staff) for new ideas from Lab² members in the future. Secondly, we do not want to overpromise, but rather prioritise research of the highest quality. We have three medium and long term perspectives.

First, we want to expand towards the Global South, developing countries and non-WEIRD societies as defined by Joseph Heinrich. The vast majority of experimental and behavioural evidence is based on Western societies.

Second, we want to try to develop new methods for replicability and meta-science research. For example, many-panel studies, replication of field experiments, artificial intelligence and language models, or gamification.

Third, we want to expand from economics to other social sciences. Political science, psychology, behavioural science and sociology are our closest neighbours.

These prospects are, of course, subject to the availability of support from funders and the research community. The Leibniz Association and Germany are particularly well placed to develop Lab² over the years. WZB, ZBW, DIW, RKW, ifo, GESIS are some of the Leibniz institutes with which Lab² is already collaborating.

How can the scientific community or the interested public participate in or support Lab²‘s endeavors?

LN: They can simply contact us with their ideas. They can join Lab² as individual researchers, institutional partners, or labs. They can join our crowd-science projects, seminar series, and events. They can also visit us as guest researchers to work on a project together.

What is your personal categorisation and outlook?

LN: Open Science has become a scientific standard. However, it requires the coordination of different stakeholders. Researchers should be open about their scientific practices and resources. Writing pre-analysis plans, being transparent about their own research, and sharing datasets and code in open repositories are some of the simplest ways researchers can contribute to open science.

They should not be challenged when their results are re-examined, and they should also be willing to re-examine the results of other researchers. It always helps to coordinate with the authors of the original paper. They are usually very helpful.

Journals should be open to publishing replications, and publishers should make it easier and cheaper to publish open access papers. Similarly, funders should support researchers with replication studies, open access fees and institutional waivers.

Thank you very much!

The interview was conducted with Dr. Doreen Siegfried on 14 June 2024.

About Dr. Levent Neyse:

Dr. Levent Neyse is behavioral economist at the Market Behavior group at WZB, Berlin, and at the German Socio-Economic Panel (SOEP) at DIW, Berlin. He is serving as an Associate Editor for the Journal of Behavioral and Experimental Economics.

He is the co-PI of Lab², a hub for replications and meta-science, supported by the Leibniz Association.

Contact: https://sites.google.com/site/leventneyse

ResearchGate: https://www.researchgate.net/profile/Levent-Neyse

OSF: https://osf.io/sdq32/#!