More than just copying
How Replication Games are revolutionising research and education
Interview with Abel Brodeur
How did the idea for the Replication Games come about? What inspired you to initiate these events?
AB: I remember talking to a collaborator and he said, ‘You know, I don’t want to do a replication because I don’t have the time. But if you could do a small workshop before a conference where we just work on codes of someone else, then I would be interested.’ And a couple of years later I was thinking about this conversation, when I was invited in Oslo for two seminars, one on Wednesday and the other one was on Friday. And I asked the person who invited me, ‘So what do I do on Thursday?’ And then I pitched the idea that we could do a small reproducibility workshop. Small, as in: we work on 1 or 2 papers, there’s going to be five or six people, and that’s going to be the end of that. We released a tweet and at the time we had almost no followers. Amazingly, 60 or 70 people registered for the event within 72 hours or thereabouts. And we didn’t really expect this, we had no rooms, no food. So I closed registration as fast as I could and I just improvised, basically. And I was thinking, I need to build teams, teams based on research interests, and I need to give instructions to the teams.
Do the teams play against each other? Is there a kind of competition?
AB: No, absolutely not. There’s only collaboration. And they don’t really do replication most of the times, it’s more robustness reproduction, but somehow replication games sounded better.
But it works.
AB: I guess. Everything that I tried to do at the Institute for Replication, I tried to do it in a way that sounds fun and is fun. I thought games would be interesting. I also quickly realized that we could do this elsewhere. It’s like the Olympic Games. We’re all going somewhere, we’re all going to do our thing, and then we go to a different city, a different month etcetera.
In the games you focus on robustness and recoding. Can you explain the significance of these primary approaches?
AB: When people think about replications, most would agree with the definition that it needs to be done with new data. I think that’s because people who came up with these definitions are doing experiments. As an economist, a non experimental economist, I’m also interested in reproducing and replicating observational studies. The vast majority of papers in economics are non experimental. Political science, sociology, and many other disciplines also use non-experimental methods. Public health, too, because you just cannot randomize most things. So how do you proceed when you do want to replicate studies that are not experimental? It’s not easy and it’s sometimes impossible. For instance, if you looked at a policy implemented in Germany, let’s say childcare subsidies. Do you want to replicate this with data from Norway? The childcare subisidies may be different, the policy may be different. You’re looking at something which is quite different from a conceptual point of view.
But I don’t think it would be feasible within a couple of days of work. If you take the replication package of an existing study, you can do a lot, you can do recoding, you can do robustness checks. Some teams do use new data, but the focus on recoding and robustness is simply for time consideration for the most part, and the fact that we’re going to be focusing on publicly available data. So the short answer would be that’s what most teams can do. Even though we do allow teams to use new data, the reality is that during an event and in the following days or weeks you can mostly just use the replication package and do recoding and robustness checks.
How do you go about matching the participants based on their research interests? You say there are so many people, they must come from different fields of economics.
AB: They also come from other disciplines. The reason why I want people to work with folks in their own subfield is that they know what a sensible robustness check or a sensible recoding exercise is. What is being done in finance and macroeconomics doesn’t necessarily make sense for an applied micro economist. The type of robustness checks or recoding that someone would do in a specific subfield is very different. The methodology is also different. We’ve thought of assigning people a study randomly, but it made little sense. For instance, if you send me a time series paper, I don’t even understand what they’re doing. Whereas someone doing time series might not be comfortable reproducing or replicating a study using a regression discontinuity design.
So that’s the main reason why I decided to put 50 or 70 people in teams from 3 to 5 where they can work with people that are doing similar research. And then I have a list of studies with publicly available data. If they’re doing health economics I would say: Here’s five studies in health economics, choose one. And a few weeks from now, you’re going to do the reproduction and/or replication of that study during the games.
Why does it matter in terms of making the game successful? Two things should be mentioned. The first one is simply that it’s much more fun to work on a study that you actually understand. And the second one is: you’re networking and spending the day with people in your subfield, so these people can be future collaborators.
Would you say the social aspect is the main reason for the success of the replication games?
AB: I wouldn’t say it’s the main reason why it’s successful, but I would say that it makes it much nicer. It’s fun. You can network with other people and you actually spend the day mostly talking about someone else’s research instead of thinking about your own and being stressed about it. It’s actually quite nice to look at someone else’s code and think about someone else’s design and study. Each time we do an event, I try to think of ways to make it more fun or more useful in terms of networking for people.
So what do you do to keep participants motivated throughout the day?
AB: You don’t have to do anything. Some teams will not stop when we’re done at 4.30 and I have to unplug their computer. People are very engaged because they work as a team. I think, especially after the lonely days of Covid, just being in a room with other people, working and coding together for hours and hours, thinking about what you could do, is great. It’s something that most people would never do. Usually, you build on your end, and once in a while you talk on Zoom, but you never sit and actually work on something with your co-authors for a full day. Research is quite lonely, you’re just working on your own. But during the games, it’s actually a team.
You connect with the others?
AB: All the other teams are working. Nobody’s looking at their emails. Nobody is looking at social media. Everybody’s focused. You have your teammates in front of you and you have to do everything you can in that day. You do feel like everybody else is working and you’re all in the same room, and you feel like, I have to actually do something. So it’s really easy to get people engaged. We start with a song, I do a presentation at the beginning, we go for lunch together.At the end of the event, each team has a few minutes to explain what they’ve done during the day, and it’s quite nice to hear what others have done. So there are some aspects that I’ve added over time, just to try to make it more engaging. But it’s really not necessary, to be honest.
Is there a didactic design for the replication games? Is there a structure to ensure both scientific integrity and participants’ learning?
AB: We do have guidelines, and we’re working on making them public. In some other disciplines it would not be possible, you’re not going to be able to reproduce or replicate a randomized control trial research in a different lab in one day. But in a lot of disciplines it’s possible and it would be easy to imitate what we’ve been doing. So we’re trying to put together clear documentation and guidelines to show here’s how you can do it.
The second part of the question about student training and what people get out of this? I would say that the training is a bit indirect. You do look at a replication package so you get an idea of how to do or not do it. And you think next time I’m going to write a replication package, these are the things that I actually did not like. You’re not the author and you’re looking at someone else’s research. So I would say most of the training you get is in terms of coding with other people, seeing how people code, see how people think of robustness checks and recoding. But it’s not a direct training in the sense of you’re looking at a screen and I’m going to teach you things.
Although, we are in the process of adding potentially a day before the event in which we would train participants who are interested. We would put together educational materials which would be publicly available. But we’re not doing it yet. I think a lot of people are interested in doing replication games because it’s mostly one day in person. So if you tell them they need to be there for a full week, nobody’s going to show up. But if you tell them you need to work before and after, but most of it happens that day, then people are willing to come. If we make it two or three or four days of training, they don’t do it. But I think, optional training the day before, or even something that’s recorded that you share with participants weeks in advance, that’s something some people would be interested in.
How do you deal with the different levels of experience of the participants in coding or research? Are there specific didactic approaches to support both novice and experienced researchers?
AB: : Usually, you need to be a PhD student or faculty or a researcher at an institution, like the World Bank or IMF or United Nations or such. So, that’s the level we’re expecting. We do not really ask any questions about their expertise in coding. We do ask what’s their favorite statistical software. That’s really important. I try to match people and also watch out that the paper that they choose will be in a language they’re comfortable with. We do encourage people who’re not very good at coding to show up and to participate. The problem is that they don’t want to do it. Especially senior people. I’m talking about full professors who are sometimes intimidated and don’t want to participate in the games because they haven’t been coding for years or decades. They have students or research assistants who are doing all the coding for them. And I tell them, no, you should participate, and it would be very interesting for PhD students to be on your team. Because you can think about potential robustness checks, you can think about specific recoding exercises and you can help writing the report at the end, whereas they do the coding.
Do you get a wide age range in these games or are there mostly younger researchers?
AB: It’s mostly students, postdocs and assistant profs. I would say right now 50% are PhD students. We do get a senior researcher once in a while. I cannot have one per team, but it would be ideal to have someone who’s actually not coding. So we do have a variety in terms of age, we’ve had someone participate who was retired. But I would say most people who participate are juniors and interested in coding.
Do you have plans to expand the scheme, by scaling up the number of the games, of participants, or by including more disciplines?
AB: Yes, yes and yes. More disciplines? Absolutely, one hundred per cent. I cannot say it now, but there should be an announcement very soon that we’re going to add more disciplines. In terms of more participants? Yes, we’re on pace this year for 700, 750 participants. My guess is that next year it’s going to be over 1,000. In terms of locations, well, we already have 15 games confirmed for 2024.
Yes, I have seen it on your website.
AB: I guess we’re going to have at least 20 for next season. It’s just happening because I receive emails almost every week from someone saying, hey, could we do games at my institution? So now it’s more turning down people than yes.
Last question. What advice do you have for young researchers or maybe also students who are considering participating in future replication games?
AB: I would say the main reason to do it is because it’s fun and you get co-authorship to a meta paper. The incentives are: you’re going to network, you’re going to actually see other people coding. And the cost is quite small. You’re not going to be working on this for six or even three months. It’s going to be reading the paper in advance, checking the replication package in advance, talking or emailing with your teammates about what to do on the day of the games. You need to write a report in the days, weeks or months afterward. That’s not a lot work. We provide a template that you can use, to make it as easy as possible. I think the main reason why you should do this is: You’re going to be able to put yourself in the shoes of someone else. What I mean is that usually you’re the author, you’re the researcher, but now you’re a replicator. You have the data and the codes and you can play with it.
Currently I don’t think there’s a structure or the incentives for writing replications. People do it sometimes as coursework or just having a look at someone else’s code. But here, with the games, there’s actually some incentives. There’s a structure for doing it. It can be fun.
Thank you!
About Abel Brodeur:
Abel Brodeur is an Associate Professor in the Department of Economics at the University of Ottawa. He completed his Ph.D. at the Paris School of Economics from 2011 to 2015 and participated in the European Doctoral Programme at the London School of Economics from 2012 to 2015.
In the editorial field, Brodeur has served as guest editor for special issues on reproductions and replications in the journals Economic Inquiry and Research & Politics. The issues covered the 2023-2024 and 2022-2023 academic years, respectively.
In addition to his teaching and editorial roles, Brodeur is the founder and chair of the Institute for Replication (I4R). This organisation is dedicated to promoting the replication of academic research as a practice to ensure the reliability and validity of scientific findings. He is also co-director of the Ottawa Applied Microeconomics Lab, another venue that aligns with his research interests.
His academic and professional activities reflect a focus on the issues of replication and research integrity, particularly in the field of economics.