Replication Games: A Collaborative Research Initiative: The Replication Games is a one-day event that brings researchers together to collaborate on replicating papers published in high-ranking journals. Replication is a crucial aspect of scientific research, ensuring that results are reliable and reproducible. By participating in the Replication Games, you will not only contribute to the integrity of research in your field but also have the opportunity to network with fellow researchers and develop your coding skills.
For the Replication Games, we are looking for professors, post-docs, graduate students students and more broadly researchers. See here for a list of upcoming Replication Games.
What to Expect:
Researchers participating in the Replication Games will join a small team of 3-5 members with similar research interests. Teams may either conduct a robustness replication, which is the ability to duplicate the results of a prior study using the same data but different procedures as were used by the original investigator - or recode the study using the raw or intermediate data. Teams may also do a combination of robustness/recoding.
Before the event, participants are expected to read the paper and familiarize themselves with the replication package. Teams need to develop a game plan outlining each team members' responsibilities during the Games. During the event, teams will work together to replicate the study. Games typically start at about 8:30am and end at 4pm local time. You are expected to work with your teammates during that time. After the replication games, teams then have to write up a short replication report summarizing their efforts.
Participants will be granted co-authorship to a meta-paper combining a large number of replications. Participants will be matched based on research interests, and a study from a leading journal will be assigned to each team. Some teams elect to submit their replication to a journal, allowing for the possibility to get another publication.
Registration to Replication Games: To register for an upcoming Replication Games event, email the following information to Abel Brodeur at abrodeur [at] uottawa.ca: (i) your field of study and research interests, (ii) preferred statistical software, (iii) the specific games you are interested in attending and (iv) if you want to participate in-person or virtually. I4R will assign you to a team based on your research interests. Pre-defined teams are also welcome to participate.
Virtual Participation: Virtual participants are welcome and should collaborate with their teams using appropriate tools and communication platforms. Further instructions for virtual participants will be provided upon registration.
Organizing Replication Games: If you are interested in organizing a Replication Games event at your institution, please reach out to Abel Brodeur at abrodeur [at] uottawa.ca for more information and guidance.
FAQ
(1) Are the Replication Games competitive, with prizes? No, the main goal is to foster collaboration and learning while replicating a study with teammates. This is a great opportunity for young scholars to meet new collaborators and develop coding skills.
(2) How much fun is it? You would not believe how much fun it can be. We've had tons of researchers participating in multiple Games. We've had participants attending from all over the world. We usually go for drinks/dinner after the Games, allowing participants to further network. For students: one thing to keep in mind is that you will have the opportunity to code for about 8 hours with other students and faculty members. This is such a unique opportunity to develop your skills as an applied researchers and meeting people in your field.
(3) Who can participate? Anyone can become a replicator, including graduate students, post-docs, junior and senior faculty members, and researchers from other institutions such as the World Bank or Federal Reserve.
(4) What are the goals of the Replication Games? The primary objective is to test whether the selected study's main results are replicable and robust. Participants should not approach the event with the aim of disproving the original study.
(5) Can participants remain anonymous? Yes, you can remain anonymous while still being granted co-authorship in the meta-paper.
(6) How much work can be done in a full day? A LOT! You won't be spending your day on social media/youtube or chatting during a long coffee break about your favorite football team. Instead, you will be spending a full day replicating a study which you've read in advance (and are familiar with the replication package). You will work with 3-4 teammates and all have different tasks to accomplish. The amount of work being done during previous Games is astounding; some teams downloaded the raw data and recoded everything from scratch; others conducted several robustness checks; some teams created or merged new variables; a few teams uncovered major coding errors, etc.
(7) How is the study to be replicated assigned to us? You will be able to select which study to replicate from a small list of studies that fit your research interests. The choice of which paper to replicate is very important. Choose a study that aligns with your team's research interests and expertise. Avoid selecting a study that uses unfamiliar methods or involves data in a language your team cannot read. It is recommended that teams meet beforehand to discuss and decide on the most suitable study.
(8) What are the post-games expectations? After the Games, teams should finalize their replication and submit a short replication report. Templates and additional information can be found here. A pre-game meeting will be held to answer questions and provide guidelines for the report.