Replication Games


We are looking for professors, post-docs, PhD students and more broadly researchers interested in a one-day replication challenge. See here for a list of upcoming Replication Games.

Participants will join a small team and being asked to replicate a study in their field of interest. Teams may either conduct a robustness replication, which is the ability to duplicate the results of a prior study using the same data but different procedures as were used by the original investigator - or recode the study using the raw or intermediate data. Teams may also do a combination of robustness/recoding.

Participants are asked to read the paper and familiarize themselves with the replication package prior to the Games. Teams should have a gameplan for the Games. Each team member should know what they are supposed to do during the Games. Teams then have to write up a short replication report summarizing their efforts.

Participants will be granted co-authorship to a meta-paper combining a large number of replications. Participants will be matched based on research interests, and a study from a leading journal will be assigned to each team. Some teams elect to submit their replication to a journal, allowing for the possibility to get another publication.

Virtual participants are welcome. Games typically start at about 8:30am and end at 4pm local time. You are expected to work with your teammates during that time.

I4R will assign you to a team of about 3-5 participants based on research interests. A group of researchers may come as a pre-defined team, but this is not required.

Interested researchers or teams: please email (i) your field of study and research interests, (ii) preferred statistical software, and (iii) which Games you are interested in to:

Abel Brodeur
abrodeur [at] uottawa.ca

Organizing Replication Games: Please reach out to us if you are interested in organizing Games at your institution.

FAQ

(1) Does `Games' mean there will be a contest and prizes? No, there are no prizes nor competition across teams. The main goal is to have fun and spending the day replicating a study with teammates. This is a great opportunity for young scholars to meet new collaborators and learn about coding.

(2) How much fun is it? You would not believe how much fun it can be. We've had tons of researchers participating in multiple Games. We've had participants attending from all over the world. We usually go for drinks/dinner after the Games, allowing participants to further network. For students: one thing to keep in mind is that you will have the opportunity to code for about 8 hours with other students and faculty members. This is such a unique opportunity to develop your skills as an applied researchers and meeting people in your field.

(3) Who can participate? Anyone can become a replicator. For the Games, we've had master/PhD students, post-docs, junior and senior faculty, and researchers at other institutions such as World Bank/Federal Reserve.

(4) What are we trying to accomplish? Testing whether the main results replicate/robust and writing up a short report summarizing all your work. Your objective is NOT to show that the results do not replicate. Your objective is to test whether the results replicate.

(5) Can I remain anonymous? We disseminate through our discussion papers each team's replication report. This being said, you may remain anonymous and still get coauthorship to the meta-paper.

(6) How much work can be done in a full day? A LOT! You won't be spending your day on social media/youtube or chatting during a long coffee break about your favorite football team. Instead, you will be spending a full day replicating a study which you've read in advance (and are familiar with the replication package). You will work with 3-4 teammates and all have different tasks to accomplish. The amount of work being done during previous Games is astounding; some teams downloaded the raw data and recoded everything from scratch; others conducted several robustness checks; some teams created or merged new variables; a few teams uncovered major coding errors, etc.

(7) Do you assign us the study to be replicated? You will be able to select which study to replicate from a small list of studies that fit your research interests. The choice of which paper to replicate is very important. Avoid choosing a study using (i) methods you are not familiar with, (ii) use super computer or very long running time, (iii) use a data set in a language none of you can read. We strongly recommend having a meeting in which you discuss which study is the best fit for your team.

(8) What needs to be done after the Games? Teams can wrap up their replication and then need to provide a short replication report. We provide teamplates and more information here: https://i4replication.org/reproducibility.html. We will also do a pre-game meeting with participants to answer questions and provide clear guidelines for the report.