Business studies needs a discussion about quantified uncertainty

Marko Sarstedt talks about his experience with Open Science

What was your first contact with Open Science?

MS: I first came in touch with it during my first years as a professor, around 2014/15, when I became aware of the replication crisis in psychology. In marketing studies we’re pretty close to the methods of psychology and I was asking myself if this crisis would also prove relevant to marketing. In the context of this discussion about the replication crisis in psychology, I recognised that Open Science is an essential element firstly for verifying the replicability of studies, and then to ensure the validity of results in the long term. I think we in business studies need an awareness for a paradigm change in research, the kind we have been seeing in the natural sciences for some time.

What is the difference between metrology and methodology? Which principles from metrology do you want to transfer to the social sciences?

MS: Metrology is the science of measurement in the natural sciences, especially in physics and engineering. These disciplines address almost exclusively directly observable matters in their research questions, for instance temperature, weight or size. Such attributes are directly measurable, at least in principle. Everything that deals with behavourial sciences has also measurement at its focus. But here we talk of psychometrics which address the measurement of non-observable, rather soft concepts. Among these are perceptions, attitudes or intentions. In our research we try to transfer the principles of recording errors that can arise during a measurement from physics to the social sciences.

Can you give an example?

MS: Measuring the weight of an object depends not only on its actual mass, but also on the geographical location of the measuring station. With increasing altitude above sea level, the air pressure decreases and influences the measurement. I must take such exterior factors into account. Something similar happens in the measurement of attitudes, but the measurement uncertainty is much more difficult to grasp. A biased sample, the use of certain questions or answering categories are influencing factors which make any measurement uncertain. You can quantify this in principle, although it is a huge task. If the measurement uncertainty is included in the consideration, the estimation will become less precise because the interval, into which the real value falls, usually becomes much wider. And this, we think, is one of the main reasons for the replication crisis. The problem is that measurements as we currently take them are much too optimistic. We assume that a measurement involving male subject A at time X is comparable to a measurement involving female subject B at time Y, without really taking into account the different contexts in which the data have been collected.  

Would you say that in an ideal world the social sciences would have a metrology like the PTB (National Metrology Institute for Germany) for physics?

MS: Yes. We published an article in Nature Human Behaviour where we demand exactly this and document the necessary steps. There must be institutions, like the Many Labs, which carry out replications in different contexts with the goal of developing standard measurements. The demand has been around in mitigated form for some time, but nothing has really happened in this direction. For example, it has been criticised  for quite some time that there are different definitions of latent concepts or that different items are being used for each measurement of a concept. But what do we learn from this? And how do we qualify the uncertainty that comes with it?

What’s the reaction to the idea of standardised measurement in marketing research?

MS: There are many researchers who have recognised the problem and the usual answer is: we definitely need standardised lists, checklists for submissions to journals like the ones used by Nature or Science. That is certainly an important aspect. But uncertainty won’t go away even with checklists. If we really want to get a grip on measurement uncertainty, we need to quantify it. In methodology journals we are very welcome with this demand and the necessary approaches for it, and we recently published a few articles about this. But it is not yet accepted in empirical-applied research. Here, “business as usual” is still king – results have to be spectacular or surprising to be funded. In my opinion, the exact opposite is needed, a focus on core effects that are properly replicated. In some sections of marketing the discussion around replications is visibly gaining traction, and some of the results are devastating. Let’s take a popular marketing concept like nudging: a recent large-scale in PNAS showed that the effect essentially doesn’t exist. This can’t go on, we have to return to fundamental effects in our field.

Does business studies have a young avantgarde aiming to revolutionise the science?

MS: The revolutionary spirit is not much pronounced in business studies, which is actually not that awful. Many researchers have recognised that there is a replication problem and first developments are already visible.There are more and more journals who request analysis scripts and make them publicly available. And I also see that it is rewarded in reviewing processes if data are made freely available at OSF, for example.

How would you like to take your engagement for metrology to another level eventually?

MS: We are going ahead gradually by involving researchers from other disciplines, for example neuroscientists who are more at home with calibration themes. Right now we are working to translate calibration approaches from the natural sciences into the social sciences. It would be nice if we could establish an institute at the end where we could study these questions systematically with metrology researchers and could talk through important concepts from business studies.

Are there other Open Science topics that you address in your research?

MS: In another Open Science project our team addresses the question of how wide-spread corresponding practices are in certain methodological concepts, such as structural equation modelling. Do authors make their data available or do they at least report correlation matrixes that allow the replication of their results? So far, the results from our analyses of several hundred studies are rather disillusioning.

What relevant Open Science practices do you actually use?

MS: If the studies allow it, we upload all data to OSF and try to cast the scripts into Shiny Apps. Those are easy-to-use R-based programmes that enable even those who can’t write code themselves to reproduce and enlarge our results. We also pre-register our studies, at least in consumer behaviour research. Of course, methodological research, which is about the development and evaluation of algorithms, doesn’t lend itself to this.

You make the findings from your research available to experts outside academia. Did this provide positive impulses for your own work?

MS: Yes, I’ve been doing that for some time. I am the only scientist on the board of the German Marketing Association and responsible for science communication. The impulses are extremely varied. On the hand I get a lot of input about relevant practical problems that we can pick up on in academia. At the same time I provide a lot of impulse for practice, for examples in the monograph series Science meets Practice, where we try to translate the often cumbersome research findings for practitioners. I also get a lot of requests for interviews, which is good for visibility.

In one of your interviews you said that marketing divisions in companies should know more about the scientific process. How do you communicate this process interestingly?

MS: I try to give an insight as entertaining as possible into individual studies that I consider to be relevant to practitioners. And I try to show how much meticulousness is spent on isolating certain effects. Let’s take an example: several years ago we had a research project about the use of fragrances in service environments where we carried out four experimental studies in total, three of them in the field and we took control groups into consideration. That shows very quickly what the difference is between a scientific study and some ad hoc marketing research. To keep your audience happy you need to present the content entertainingly, and occasionally you need to be pragmatic about it. But it’s something yo can learn, for instance if you take part in science slams.

How important is Open Access to you?

MS: As a matter of fact, it’s a criterium for targeting a journal. Of course the paper must be a thematic fit for the journal, but if it’s otherwise a  fit, we’d rather send the paper where it can be accessed freely by readers – if it is accepted. The DEAL Agreement has been a decisive factor, because suddenly a number of journals became Open Access without additional fees. My willingness to pay for Open Access publications is lukewarm, to say the least. Publishers make enough money out of us at it is.

Do you also try to go to so-called scholar-led-journals?

MS: Yes, but then, as a professor with civil service status I can afford to. Many of the young researchers don’t have this luxury and I can fully understand why they do it differently. If we can choose between several journals and if there is one that’s not so highly-ranked, but Open Access because of the DEAL Agreement, then that’s where we send the article.

Do you have tips for young researchers how they can enter into Open Science?

MS: : I would assume that most business economists have a certain affinity to psychology. So my tip would be to look at the discussion and editorial practices there. My second tip would be to look at OSF and GitHub. I think those are the central anchor points for Open Science in business studies. At the end of the day, the core element of Open Science is to make your data and analysis codes available and to make the entire research process transparent.

Is Open Science a familiar concept in business studies?

MS: I can’t speak for the entire discipline, but it is familiar in my network and practised more and more.

What is the biggest additional benefit in the context of Open Science for you?

MS: As scientists we are in a privileged position, we can do research in areas that interest us and the general public pays us for it. Therefore the general public has a right to be told what we’re actually doing. And Open Science creates this transparency and therefore a certain legitimacy for what we do. At the same time, Open Access has many positive effects for me, for example when I look at the citation numbers which are significantly higher for those articles. Open Access also has an important intrinsic value because scientists from less well-equipped universities or from other countries, where access is much more restricted, can participate in the science system.

How could more business economists be persuaded of the benefits of Open Science?

MS: I think this is in the responsibility of the more senior researchers among us, to take Open Science aspects into PhD training and make it an integral part. But I don’t know of any initiative yet to establish a dedicated Open Science course in a PhD programme. I believe this is needed. What’s p-hacking, what’s HARKing, and how can such manipulations be identified? What’s the contribution of Open Science in this? Those are topics for a good PhD programme. It’s also important to explain why such practices are probelamtic, because in my experience some researchers have no problem awareness in this context.

There are many people all over Germany who establish PhD programmes and summer schools, but they all work for themselves at their own institution. Do you have a network here?

MS: The Open Science Initiative at LMU is doing exactly this, involving various institutes and chairs. The LMU Open Science Center regularly offers workshops and lectures on Open Science topics. Unfortunately, I am not aware of any Germany-wide initiative which explicitly addresses Open Science in the context of marketing.

Thank you!

Dr Doreen Siegfried conducted the interview on 14 November 2022.

About Professor Marko Sarstedt

Professor Marko Sarstedt heads the Institute for Marketing at the LMU Munich School of Management. He is also professor and honorary doctor of Babeș Bolyai University Cluj in Romania. His research focus is on the improvement of multivariate analysis methods and metrological approaches to a better understanding of consumer behaviour.
The publications of Professor Marko Sarstedt count among the most cited papers in the social sciences with more than 100,000 citations (Google Scholar). He has won a large number of awards and prizes, among them five Emerald Citations of Excellence Awards and zwei William R. Darden Awards.

Professor Marko Sarstedt discusses his research findings in leading print media such as Die Zeit, Huffington Post, and Spiegel, and talks about them in TV documentaries on ARTE or MDR and also on the scienbce platform Latest Thinking. In the 2020 F.A.Z. Ranking of the most influential German-speaking economists Professor Marko Sarstedt occupied second place in research. Marko Sarstedt is a member of “Clarivate Analytic’s Highly Cited Researcher List”.

Contactt: https://www.marketing.bwl.uni-muenchen.de/team/professoren/prof_-dr_-marko-sarstedt/index.html

ORCID-ID: https://orcid.org/0000-0002-5424-4268

Twitter: https://twitter.com/MarkoSarstedt

LinkedIn: https://www.linkedin.com/in/marko-sarstedt-06819a82/?originalSubdomain=de

ResearchGate: https://www.researchgate.net/profile/Marko-Sarstedt

to Open Science Magazine