Funding
22 June 2023
We are grateful for the funding provided by Open Philanthropy! Here is our proposal.
Brief Statement: Which scientific results can we trust? We plan to push forward the boundaries of what we know about scientific reproducibility in general, and significantly expand the number of studies being replicated in economics, and related fields.
Description of the Project The project's main objectives are the following:
1- Generate evidence by systematically reproducing (i.e., push-button/running the codes) all empirical results from studies published in several leading journals and replicating (i.e., recoding using raw data or sensitivity analysis) hundreds of studies from these outlets. For economics, our collaborators (i.e., data editors) and staff are currently reproducing all studies published in eight leading outlets. We are also reproducing and replicating studies from leading outlets in other fields such as political science. As of January 2023, we have over 100 ongoing or completed replications, and reproduced hundreds of studies. This was accomplished without any funding. We have two streams to generate replications. First, our board of editors (https://i4replication.org/people.html) suggest names of replicators, which the PI reaches out to. Second, we organize monthly Replication Games all over the world (https://i4replication.org/description.html). The Replication Games are a one-day event in which teams of 3-5 researchers replicate a study. In terms of incentives, all replicators are given co-authorship to a meta-paper combining all replications. a. Funding from OP would allow us to replicate all studies with a detailed replication package published in the American Economic Review, excluding studies using proprietary data for the main analyses. We would also replicate significantly more studies from other leading outlets. Our objective is to reach 300-500 economic studies replicated per year by 2026. b. Funding from OP would also allow us to help journals reproduce significantly more studies.
2- Development and increased access to educational material for replication, building capacity to conduct replication through training, support, and a growing network of advocates and allies. We have developed guidelines for reproducibility and replication over the past 18 months. For instance, we developed a template to be used by replicators: https://osf.io/8dkxc/. We also worked with the Berkeley Initiative for Transparency in the Social Sciences to develop the Social Science Reproduction Platform: https://www.socialsciencereproduction.org/. These tools allow to better coordinate and standardize reproductions and replications. a. Funding from OP would allow us to scale-up Replication Games and allow researchers/students from minority groups and low-income countries. These events allow students to work with more experienced researchers in their field and to network. We also plan to have an extra day of training for students attending the Replication Games. This training will delve into open science tools and software training.
3- Strengthen the scientific ecosystem by enabling authors, replicators and journal editors to effectively and equitably change norms at scale through open science policy and protocol development. Over the past 18 months, we have put together three special issues dedicated to replications (all co-edited by the PI), and helped many journals develop “replication section”. See here for more details: https://i4replication.org/publishing.html. a. Funding from OP would allow us to develop guidelines to help editors and original authors better edit and respond to replications/comments. We would also like to put together more special issues dedicated to replications and work with the American Economic Association to put together a journal dedicated to replications.
This project would start in July 2023 and end in December 2025.