Open Science enables structuring and transparency

The three key learnings:

• Practices such as the pre-registration of studies and the sharing of data and materials via platforms such as the Open Science Framework (OSF) not only improve the quality of scientific work, but also counteract publication bias.
• The discussion of publication bias and research errors in the context of teaching prepares students for transparent research practice and motivates them to apply open science principles in their own work.
• Active participation in Open Science and the application of its practices, such as the organisation and sharing of research via the OSF, not only supports individual research work, but also contributes to the scientific community by providing improved access to research results and materials.

Could you please start by briefly explaining your research topic?

Susanne Adler: My first research area is dedicated to consumer behaviour, i.e. the question of why we make certain decisions. The second area focusses on meta-research with the aim of evaluating and improving research processes. A central concern here is not only to collect data carefully, but also to ensure that it can be used sustainably. Much potential remains untapped, as information on gender, age or time-of-day effects, for example, which are collected incidentally, are often not analysed because the data was primarily collected for another research objective. In addition, control variables that show no effect are often not reported. These observations motivated me to take a closer look at this area. Interestingly, the corona pandemic, which coincided with an early phase of my PhD, gave me the opportunity to focus more on these meta-topics, as some laboratory experiments were not feasible.

How did you become interested in Open Science?

Susanne Adler: My approach to Open Science was primarily characterised by early influences in my academic career. A friend of mine who was doing a PhD in psychology was working with me on a research project. He suggested registering our project on the Open Science framework from the beginning, including pre-registering our study and publishing all the data and writing the R code before we even started collecting data. This approach made sense to me, and I was sort of introduced to the practices of Open Science. Additionally, through my own research in consumer behaviour, I became aware that there is a significant publication bias, often associated with small sample sizes and small effect sizes. I began to realise this during my bachelor’s degree when I noticed that the empirical results from undergraduate projects were not as convincing as the studies published in academic journals. Initially, as a student, I thought this was solely due to my own inexperience. Over time, however, I realised that selective reporting in specialist journals is also a problem.

Could you give us some examples of best practice from your day-to-day work that are particularly close to your heart and that you would like to share?

Susanne Adler: There is no universal best-practice example that works perfectly in every situation. For me personally, however, it is very important to be able to work with data whose origin and collection methods are transparent to me – beyond the extent described in the corresponding publication. I consider the pre-registration of studies to be another important building block. For me, these aspects together form an overall package that makes a study trustworthy and transparent and enables me to build on a solid database.

So now we’re talking about how it’s possible to understand the work of others, right?

Susanne Adler: That’s right. Transparency in research is not only important to me in the work of others, but also in my own projects. In concrete terms, this means that I carry out pre-registration for confirmatory research work. As soon as I have data, scripts, questionnaires or other stimuli, I make them available via the Open Science Framework (OSF). I make sure that the materials are made available in a form that I myself consider useful. A crucial point is that there is always sufficient information on the data stored on OSF. This includes, for example, clearly explaining the data collection methods and the research process as well as a codebook that explains the variables. It is important to me that the data is not just a series of numbers, but that the context and meaning of the variables are also clear. The possibility of traceability is essential, even if not every detail can and must be perfect. For example, I understand if certain functions in an R code no longer work properly after some time. Of course you should write stable code, but getting involved as a researcher in the first place and making materials accessible is more important than perfection. It is crucial that the research community can at least understand the basics of the work.

Have you already realised personal benefits from using various Open Science practices?

Susanne Adler: Yes, definitely. In particular, I am often asked about our results reports, which are written in Quarto documents or interactive web apps. This also makes my work accessible to those who may not work with R. Especially in the research community, I get questions like “How exactly did you implement this and is it worth it?”. In such cases, I like to refer to materials that I make available on the Open Science Framework (OSF). For example, I then say: “Look, all the code is on the OSF and is very easy to reuse. It would be great if you could give me feedback if you use it and encounter problems or have suggestions for improvement.” This openness not only enables an exchange about research practice, but also emphasises that research is fraught with many uncertainties and is rarely as straightforward as it is sometimes portrayed.

You have already mentioned that you make intensive use of the Open Science Framework. Does this mean that you publish all your research data, scripts and other materials there, or do you also use other platforms?

Susanne Adler: A large part of my work is on the OSF wherever possible. It is particularly important to me that all components of my research – from the methodological materials to the scripts to the results reports – can be collected centrally and given a DOI. This provides a permanent link to all resources. The platform allows me to organise my projects effectively and makes it easier to share them with others. I also benefit from a network in which the added value of OSF is widely recognised. Projects that have not yet been finalised initially remain private in OSF and are not visible to outsiders. As soon as a project is published as a paper, I also publish the OSF project.

Is it common practice in consumer research to share research data and materials, and is this considered good practice in your community?

Susanne Adler: Whether it is common practice to share research materials and data depends very much on the respective research area within consumer research. From my perspective, too little material is shared overall. In the research community in which I am active, however, it is considered good practice to at least make the data available. However, sharing data and materials also offers the advantage that all resources are centralised and not just stored on personal storage media such as Dropbox or private computers. This is particularly important as researchers, especially doctoral students, may leave the academic world. This ensures that all materials remain accessible in the future.

Suppose another doctoral student is about to decide to use Open Science Framework (OSF) for her work and asks you for advice. What main advantages of OSF would you emphasise, especially in relation to your own research work?

Susanne Adler: I would definitely recommend using it. The decisive advantage of OSF lies in the structuring and organisation of research data and research materials. It requires me to prepare my data in such a way that it is understandable both for myself and for outsiders. This not only promotes clarity in my own work, but also facilitates the sharing of data within the project team and with the research community. OSF is therefore not just a repository – it improves accessibility, traceability and collaboration. In addition, OSF is user-friendly, which simplifies handling and provides efficient support.

You have emphasised the advantage of the Open Science Framework for structuring your own work and praised its simple user interface. So would you say that using OSF can also be seen as an investment in your future self?

Susanne Adler: Definitely. Structuring-whether on the OSF or in your own filing system-is a long-term investment in your own work. There were times, especially at the beginning of my dissertation, when I created folders whose structure I can hardly understand today. In fact, I know the situation where projects or data are left lying around for months because, for example, more urgent tasks took priority. With data analyses in particular, it can quickly happen that you ask yourself: “What exactly was I doing here?” In such moments, a clear data structure proves to be a valuable tool for self-organisation. It forces you to create a structure that not only you can understand in a few years’ time, but that is also accessible and understandable for others.

In the context of replication studies and the importance of reproducibility and replicability of research results – have you already gained experience in conducting such studies?

Susanne Adler: Yes, I have actually already carried out replication studies and last summer semester I even led a seminar at the LMU in which student groups implemented replications. The experience was very enriching for the students, as they read scientific articles in great detail and with a more critical eye. Normally, we consume articles quickly and focus on the core messages. However, replication demands a deeper engagement with the material, often revealing what is missing in the reporting. Discussions often arose about exactly how certain methods were applied, which pointed to ambiguities in the original studies. These experiences were also instructive for me in terms of reflecting on how I can present my own research findings in a way that makes them understandable. One challenge, for example, is the use of specific software, such as Qualtrics for surveys, which not everyone has access to. A PDF version of the questionnaire can offer an approximation, but does not correspond to the original, which is a compromise.

In your work in the field of consumer research, you mentioned that you work together with psychologists. How relevant is it for you to deal with the methods of psychology? Does this have an influence on your research?

Susanne Adler: The topic is extremely relevant and interesting for me. Many of the theories used in consumer research originally come from psychology. In a way, large parts of consumer research can be seen as a form of applied psychology, in which psychological theories and methods are adapted and applied. Even though consumer research encompasses other areas, the field in which I work is strongly influenced by psychology. Therefore, the orientation towards the principles of Open Science plays a particularly important role in my work.

Could you give us an insight into meta-research, particularly with regard to the reproducibility and replicability of research results? What insights have you gained in this area?

Susanne Adler: In terms of reproducibility, it is clear that many details in scientific articles are not fully reproducible. Details are often missing, which is due to the word and character limits in journal articles. In my view, not all details need to be described in the research paper, but it should be ensured that the research remains reproducible. One way to do this is through Open Science, for example by sharing materials such as questionnaires, analysis codes or the publication of additional documents such as literature reviews, which then describe their methodology in detail, including keywords and approaches used, or documents in which any deviations from preregistration are accessible. While the practice of preregistration is considered standard to make research plans transparent, changes often occur during the course of research. In such cases, deviations should be documented and additional analyses, such as sensitivity analyses or robustness tests that were not preregistered, should be performed.

Consumer research offers fascinating insights that can be of interest far beyond academic circles. Do you use different channels or formats to make your research findings accessible to a non-scientific audience?

Susanne Adler: I have actually written a few articles for the practice and worked on a book project in the “Science meets Practice” series, which deals with the multisensory effect of sales environments. In addition, I have participated in a few science slams where I humorously highlighted the challenges and pitfalls of the scientific process. In one of my talks, entitled “The 1000 deaths of scientific knowledge”, I presented the path from brainstorming to publication – with all its hurdles and pitfalls – accompanied by a variety of memes. Although the lecture primarily dealt with scientific topics, it met with broad interest, as the audience could easily relate to the experiences of setbacks and unfinished projects.

What advantages do you see in not only imparting knowledge, but also receiving feedback?

Susanne Adler: A direct response is often the positive feedback on my presentations and the audience’s identification with the problems addressed. Students in particular feel encouraged when they learn that research is not perfect, but also involves uncertainties and not always clear results. These conversations make it clear that science is a process with ups and downs, just like in any other profession. By openly discussing these aspects, I promote an understanding of science as a field full of discoveries and challenges. This exchange is also informative for companies, as it shows that research results are not always directly transferable, but can still provide important impulses for practice.

Do you also publish results that are not statistically significant?

Susanne Adler: We recently published an article in “Marketing Letters” in which we looked at the influence of caffeine on the attraction effect when selecting several products, a recognised phenomenon in consumer research. We observed this effect in the first two studies. However, in a third, methodologically modified study, in which we reduced the caffeine dose, among other things, we were unable to replicate the effect. We were faced with the challenge of how to deal with these results in the article, especially as non-significant results are rarely published in consumer research. We decided to present the results transparently and discuss the methodological differences. The article was accepted, which we were pleased about. We were concerned that the paper might be rejected or that we would be asked to re-prove the effect of the reduced caffeine dose to be sure, which is not trivial due to the effort and cost involved.

I would like to ask for your personal outlook. As someone who has already gained valuable experience and is still at the beginning of their scientific career, what advice would you give to young doctoral students who are just starting out, or even more experienced scientists who have not yet dealt with Open Science? What first steps towards Open Science do you recommend?

Susanne Adler: I would recommend the “Student’s Guide to Open Science” by Charlotte R. Pennington, a compact handbook of around one hundred pages. I think this guide is an excellent introduction to open science. Given the numerous developments that have taken place both in psychology and in other disciplines, Open Science offers a wide range of opportunities for engagement and contribution. This can seem overwhelming at first. My first piece of advice would therefore be to read this book to get a basic overview. As a second step, I recommend choosing a specific project that seems interesting to you. This could be creating an OSF project or working with R code for the first time instead of traditional point-and-click software. Ideally, you should work with someone who already has experience so that you can learn from existing structures. The third tip is to follow the activities within the open science community. You don’t have to actively participate right away, but it is helpful to be present in Slack channels, for example, and observe what initiatives are going on. Twitter is also useful for finding out about current projects. This gives you an overview of the possibilities and allows you to get involved in a targeted manner if necessary.

What initiatives are there to promote Open Science in business studies in particular?

Susanne Adler: Business administration poses a challenge in this context. My experience lies primarily in the field of psychology and meta-research, particularly with regard to the issues of replicability and reproducibility of research results. One notable initiative is the campaign launched by Nature on the reproducibility of its articles. In the field of business administration and consumer research, the Institute for Replication, which was founded by Abel Brodeur and is known for its “Replication Games”, is particularly worthy of mention. This initiative also offers interesting starting points for business administration and consumer research, both in terms of content and methodology. I am currently involved in a multilab replication study on Construal Level Theory, for which I was responsible for the translation of the German questionnaire. This initiative, coordinated by a Swedish research team, particularly excites me because of the significant effort involved and the long duration of such projects. The project has been running for more than three years and has been accepted for publication by Advances in Methods and Practices in Psychological Science. This experience motivates me to initiate similar studies in consumer research in the future, for example on topics such as the attraction effect, which is strongly influenced by methodology. Multilab studies offer the opportunity to study effects more broadly by having many institutions each contribute a part, rather than having a single institution conduct a single large-scale experimental work.

An extensive team project.

Susanne Adler: Yes, it is particularly revealing when you look at the variance of the effect sizes. If similar effect sizes are observed across different laboratories, this indicates an effect that can be easily generalised. However, if you carry out the study in, say, 100 laboratories and the effect sizes vary from negative to medium to very large, it becomes clear that the effect is subject to considerable variation.

In view of your observation that there is not yet a large community for Open Science in business studies, how do you assess the future development of this area, especially in the context of a possible interdisciplinary community of psychology and business studies?

Susanne Adler: I am optimistic that interest in the topic of open science in business studies will grow, especially among young academics. The response in my discussions has been consistently positive. The challenge that well-conceived research projects often do not deliver the hoped-for results and are therefore difficult to publish is a widespread problem. This leads to frustration, especially among younger researchers, as their career development depends heavily on the number and quality of their publications. In addition to these challenges as a junior researcher, there may be a lack of funds for extensive data collection. However, there is the opportunity to implement exciting projects by analysing existing data. For example, you could explore questions relating to processing times or times of day, i.e. topics that are often present in data sets but are rarely used. However, the problem arises again that data sets are not always accessible, which leaves potential untapped.

How do you integrate topics such as publication bias and research misconduct into your teaching and encourage your students to apply these concepts in their theses?

Susanne Adler: As part of seminars on the application of research methods, I like to integrate information on topics such as publication bias and various errors in research. This gives students an insight into this important topic. I also encourage them to apply these principles in their Master’s theses, although no one has yet taken me up on this offer. I present the possibilities for preregistration, but emphasise that it is voluntary and it is up to them to decide whether they want to implement it in their own work. Regardless of this, it is necessary for students to submit all the data collected when submitting their theses.

How do you think we can attract more researchers from the business world to Open Science?

Susanne Adler: I see two starting points for getting more researchers interested in open science. On the one hand, the conviction that transparency and the open sharing of results are essential for science. These general advantages are easy to understand. On the other hand, the implementation of Open Science initially requires additional effort, for example the detailed planning of data analyses before the survey or the pre-writing of analysis codes. Here it is effective to point out personal advantages. One practical argument is everyday experience: many researchers are familiar with the problem of not being able to find their way around their own folders after a few months or being disappointed when projects do not deliver the expected results. It is particularly valuable for early career researchers to know that even complex projects that take a long time to publish can be made visible through preprints. These can already be used in applications and could thus promote an academic career and visibility.

Thank you very much!

This interview was conducted on 5 March 2024 by Dr Doreen Siegfried.

About Susanne Adler:

Susanne Adler has been a research associate and doctoral candidate at the Institute of Marketing at LMU Munich since September 2021. She previously worked as a research assistant and doctoral candidate at the Chair of Marketing at Otto von Guericke University Magdeburg. Susanne Adler completed her master’s degree in consumer psychology and market research at the Harz University of Applied Sciences in Wernigerode.

Susanne Adler conducts research at the interface between consumer research and meta-research. Her research therefore covers two topics. Firstly, she focuses on consumer decision-making in complex environments with various contextual factors such as sensory stimuli. While individual research findings provide valuable insights into the decision-making process, it is often overarching structures that determine the direction and development of a scientific field.

Her second research topic therefore encompasses a meta-perspective aimed at a more integrative understanding and better utilisation of existing knowledge in consumer research. For example, she uses bibliometric methods to identify academic structures and latent research topics and deals with the summarisation, evaluation and integration of existing knowledge. As part of this line of research, Susanne Adler is also concerned with the reproducibility and replicability of research results and with open science.

Contact: https://www.som.lmu.de/imm/de/institut/kontaktseite/susanne-adler-1210c816.html

LinkedIn: https://www.linkedin.com/in/susanne-adler-46a69a212

ORCID-ID: https://orcid.org/0000-0002-3211-6871

Open Science Framework: https://osf.io/e8qsb/

ResearchGate: https://www.researchgate.net/profile/Susanne-Adler