Replications are the most direct way to join a discourse

Martin Sievert talks about his experience with Open Science

How did you come into contact with Open Science?

MS: One of my professors during my master studies, Professor Dominik Vogel, set much store by students understanding scientific working. He gave a course over two semesters where students go through the entire research process. In this context, Open Science was a constant theme, from the preregistration of hypotheses to making codes and data available so everything is reproducible. I intuitively thought this made sense and I continued with it later. Right now, Open Science is more a key selling point because it makes you stand out. In my view it should be a minimum standard. After I finished my bachelor studies I wrote two papers with Dominik Vogel that we preregistered with Open Science Framework. We also uploaded the data to OSF and the R-codes to Code Ocean. The research projects in my PhD thesis are basically designed the same way. Open Science practices are seen as something desirable in Mannheim, too. I think the additional effort is manageable and it makes research in general more trustworthy, I believe.  

Are there discussions in economics about seminars like the one you attended? Do you see differences in the USA compared to Germany?

MS: Absolutely. Most practitioners of Open Science pass on their knowledge to their students, maybe also want to be multiplicators within the community. At least this is very common in my environment. Something I also see: scientists who have not familarised themselves with Open Science for their own work find it difficult to integrate it into their teaching. As a consequence, some students are introduced to Open Science practices in their courses while others are not. It could be that a platform for exchange among scientists is lacking. One good idea would be if students did more replication studies while learning their academic craft. I think it is sensible if students do a replication study and can concentrate on the essentials. Ideally students would go through the entire research process in one semester before they graduate – and without the pressure of coming up with something completely new. They’re told to pick a relevant part of their area of research, to look for something that interests them, and replicate the study as well as they can with the means available. Such an approach would make for very good learning outcomes regarding scientific practice. But for this you need people ready to move ahead and willing to apply such approaches at the professorial level. In Mannheim we have the Open Science Meetup and the university library who transfer Open Science into the entire university. I think that is a great step.

Have you seen concrete benefits from Open Science? Could you showcase your Open Science activities during your research visit in the USA and gain credit for them?

MS: My Open Science activities occasioned my visit because I’m staying with the colleagues whose study I replicated. I met the primary author at my first conference and we’ve stayed in contact since. However, I replicated the study completely independently of that. I think it’s important not to by-pass the authors when you do a replication study. And the contact has been very intensive because of the replication study. There’s an obvious overlap in research interests. I’ts about the cumulative generation of knowledge and the continuation of existing ideas. In that respect my sum-up is that Open Science means visibility. In the case of a published replication you ideally discuss it with the original authors, and they will be aware of you. Most people will also have a certain regard for you because you replicate their research.

Could you verify the study’s results?

MS: I couldn’t verify the study, even although we tried several things such as equivalence tests. These are null results. That’s what makes the replication truly exciting because now I have to think about why this happened. And what does it mean for the theoretical mechanism we are studying? That’s the difficult part of the replication for which you should prepare yourself mentally. But it is also an important part and a necessary one. You should look at and be able to explain the differences you possibly found and why.

What’s your talk with the primary author like?

MS: We are now talking about the implications. I didn’t tell the primary author about the study in advance. In the future I would rather do that. I think it’s better to talk with people beforehand because you can get a lot of input, for the design for example. That minimises later discussion points and you get someone who can look at it who’s familiar with things. However, if the reaction is rather low-key, it’s not a reason to desist from doing the replication study.

How’s the interaction in the reputational cosmos?

MS: The reactions from, and the communication with, the three primary authors of the study I replicated were very different. All in all they were pleased with the attention, even if there were nuances. I am currently in contact with only one of the authors who is very responsive. Of course he regrets this outcome, because his results could have had so many good practical implications for public organisations. But in sum the reactions from the authors and the research field have been very positive. I’ve got one e-mail from a US researcher who has many publications. He congratulated me and wrote he would have liked to do the replication himself, but he thinks it’s great that the job has been done, so to speak. This also led to a personal meeting with him. All in all, the replication has brought more visibility, better networking and possible cooperations. For me it is the most direct way to join a discourse.

What’s your attitude on Open Access?

MS: I see this very pragmatically and currently it’s a unique selling point for me: papers are read and cited more often. If you have a chance to get more visibility, you should use it. And of course I want everyone to be able to read the publication.

What does that look like in practice?

MS: I look around where publication is possible, who’s included in the DEAL contracts. It’s not the first criterium for choosing a journal but maybe you put more effort into a submission to Wiley’s because you know it’s potentially an Open Access publication. In Mannheim the university library pays the charges. All we have to do is click twice, sometimes not even that. In addition, the publications and materials are entered into a project directory at Open Science Framework where they are usually also crawled by Google Scholar. We also recently published a preprint on SSRN. My impression so far: preprints lead to more discussion around the paper and hence to more visibility.

Do you publish research data and scripts?

MS: Yes, we usually publish the data. We always upload the materials, both to the journal and the Open Science Repository. By now I also use Code Ocean for analysis codes because it is easier to handle. It’s like a time capsule. I define the R version and the packages outside the code and then I create the code within this R environment. At the end I seal the code and then you can no longer change anything inside the capsule. That’s quite useful. I can click START and it will still work ten years from now.

What exactly do you upload to Open Science Framework?

MS: I do the preregistration there, I link to Code Ocean and apart from that I upload everything that I would also upload in Dataverse. That’s preregistration, script, data, and everything else needed for simplification is linked. It’s also accessed quite often. But you have to link it always and everywhere to get visibility. It goes into every post on Twitter. It’s in my CV, it’s on the website. For this OSF works very well.

Has anyone ever contacted you and complimented you because everything can be found there?

MS: Actually yes. The editor at the last journal thought it was brilliant. He had just taken over the editorship, introduced a replication section and changed a lot else. And I think it’s very useful for him to see how this works at the practical level so he can adapt his journal guidelines. Of course an editor prefers it if you have everything deposited in structured form in an OSF, instead of things lying around somewhere in Dataverse. In this journal it is implicitly required for empirical studies, and for replication it’s a prerequisite.

Let’s talk about science communication. What is the role of dialogue with non-scientists? Do you process your findings also for the general public?

MS: In selected cases, yes. The discourse around the replication study is among scholars, and I can’t see any added value there. But for a field experiment about Covid vaccinations, my co-author, Dr Florian Keppeler, has written a policy paper, published a preprint and put a lot of work into outreach. It was good to see that research can find favour at the non-scholarly level. I think you have to divide science communication well, because the work effort is very high.

Where do you as a PhD student see the development of Open Science?

MS: There are many perspectives. What comes to mind is that ideas die out if they don’t spread as quickly as possible. The science system is dominated by incentives that make it difficult to adopt such an innovation. Right now there’s a lot of idealism and pragmatism. The question is where people come into contact with Open Science. There has to be an initial spark. I think certain skills must be taught earlier if the science system is to change. Students must be enabled better to think critically, to look closely. People must be enabled to assess information critically. This includes having an understanding of what is good scientific practice, what is Open Science. It doesn’t matter if I stay on at university after finishing my business studies or if I join a management consultancy. If I want to create real added value in business, I need to be able to handle scholarly publications critically. We need lecturers who introduce Open Science into teaching.

Thank you!

The questions were asked by Dr Doreen Siegfried.

The interview was conducted on September 20, 2021.

About Martin Sievert

Martin Sievert is research assistant and PhD candidate at the chair of ABWL, Public & Nonprofit Management at the University of Mannheim. His research interests are Organisational Legitimacy, Person-Environment Misfit and Behavourial Public Administration.

Contact: https://www.bwl.uni-mannheim.de/helmig/team/martin-sievert-msc/#c77485

ORCID-ID: 0000-0002-1331-2439

Twitter: @martinsievert

LinkedIn: https://www.linkedin.com/in/martin-s-00143214b/

ResearchGate: https://www.researchgate.net/profile/Martin-Sievert-3

to Open Science Magazine