Funding
May 2022
We are grateful for the funding provided by the Social Sciences and Humanities Research Council! Here is a summary of the proposal.
Context: Which scientific results can we trust? In order to assess the reliability or reproducibility of published results in economics and other related disciplines, a number of large-scale replication projects are taking place. Replication here means that the original study’s main significant result is chosen for replication, with the study performed again on a new sample using similar methods and tests. One of the co-PIs and other collaborators led an experimental economics replication project (Camerer et al. 2016) and a social science replication project (Camerer et al. 2018). Pooling the results of these large replication projects yield a replication rate of about 50%. This low rate can be due to many factors. First, many previous studies have been performed on small sample sizes, implying low statistical power. Ioannidis et al., 2019 survey 159 empirical economics literatures and find that the median statistical power is 18% or less. Second, there are typically many ways to test a hypothesis, giving researchers many “researcher degrees of freedom” in the analysis. Decisions during analysis to make results more eye-catching could introduce bias. Specification searching (or “p-hacking”) has been found to be a problem in many social science publications (Gerber and Malhotra, 2008, Brodeur et al, 2016). Low statistical power and many degrees of freedom in the statistical analysis can give rise to “statistically significant” results that are unlikely to be true. These factors make it hard to disentangle true results from false positive and false negative ones.
Objectives We plan to push forward the boundaries of what we know about scientific reproducibility in general, and significantly expand the number of studies being replicated in economics, finance and related fields. Our focus is on non-experimental studies. The backbone of our proposal is the Institute for Replication (I4R, https://i4replication.org/).
Definition The main goal of our project is to reproduce and replicate a very large number of studies published in leading economic and finance outlets. A variety of definitions are used for the terms reproducibility and replicability. We rely throughout on the following definitions: Reproducibility: The ability to duplicate the results of a prior study using the same materials and procedures as were used by the original investigator. Replicability: (i) robustness replication, which is the ability to duplicate the results of a prior study using the same data but different procedures as were used by the original investigator. It basically involves conducting sensitivity analysis. (ii) direct replication, which involves recoding the study using the raw or intermediate data. Replicability may also involve a combination of these two types of replications.
Project Description and Methodology: The main goal of our project is to reproduce and replicate a very large number of studies published in leading academic outlets. Procedure for reproducing: For reproductions, we follow these steps for each study. First, our team (or a data editor) assess reproducibility (i.e., running the code) and standardize the file structure and code if this was not done by a data editor. This task will be done by a team of postdoctoral fellows, with the help of the PI and data editors. The workflow for each article is similar to those used by data editors. The first part consists of an assessment of the README, the manuscript, and of the package contents. We assess whether data provenance is completely described, whether the manuscript and/or the README contain data citations, and whether there appears to be a complete archive of the program. Data provenance is assessed and verified, even when data are provided. When data are not provided, data provenance is even more important, as it must completely describe how the data may be obtained. This can range from simple click-through downloads to complex application processes. The computer code and the README are assessed in terms of the software requirements. Finally, the computational resources are summarized, if reported in the README or deduced from other descriptions. This information is summarized in a preliminary report. We then downloads any necessary data, and follows instructions in the README to run the programs provided. Once the results have (or have not) been reproduced, our team completes a report. While based off of a template, the report has many free-form and narrative components, describing the steps the replicator undertook to achieve full computational reproducibility. Our team, comprised of postdoctoral fellows, is trained and mentored in how to convey steps in a concise but complete way, allowing the reader to understand fully what was done, and why it may not have worked. Postdoctoral fellows have access to a bank of frequently used "canned responses" that provide constructive feedback and a checklist of requests. Everything is then reviewed by the PI or an experienced collaborator (i.e., data editor) before contacting the original authors and making this report publicly available. The cycle described above gets repeated for every new published study in journals without a data editor (e.g., Quarterly Journal of Economics). Of note, the PI’s recent paper “Methods Matter" was chosen as the template by the American Economic Review for the procedure described above. Each replication will thus contain an open access dataset with standardized file structure and code (including variable definitions) - a novelty in economics and an innovation aimed at eliminating the fixed costs to future users. Procedure for replicating: Depending on the topic, method, data availability and funding, the study might be replicated by our team of replicators (e.g., post-doctoral students). If this is not the case, we will be reaching out to potential replicators. This step is done by our large team of collaborators (i.e., members of the editorial board) who propose potential replicators for each study that was successfully reproduced. The PI then reaches out to these researchers. The editorial board comprises about 25 top economists covering all sub-fields in economics and finance. They have an influence on the discipline through a large network of co-authors; editorial positions at top journals; they are directors of other research institutions; and have supervised hundreds of PhD students, making our influence extraordinarily broad. Replicators then conduct a robustness or direct replication. The replicator completes a report summarizing their replication effort using the Social Science Reproduction Platform (https://www.socialsciencereproduction.org/), which was developed by one of the co-PIs. Detailed instructions and suggestions for robustness checks will be provided to replicators. Our team of collaborators have developed these guidelines which are available here: https://bitss.github.io/ACRE/. Our team then send the report of the replicator(s) to the original authors, who will be invited to respond. Reports and responses by the authors and replicators are then published on our webpage and in our discussion paper series. Importantly, we will disseminate replications no matter their outcome along the original authors’ response. We also plan to organize Replication Games where researchers team up to replicate studies. I4R’s first Replication Games are happening in Oslo, October 27th, 2022: https://twitter.com/I4Replication/status/1565363821736329222. Over 60 researchers will participate to replicate 14 studies. Many top researchers have already shown interests for having the Replication Games coming to their respective institution (e.g., HEC and ENS in France, DWI in Germany).
Reproducibility Our short-term objective is to reproduce all empirical studies published in eight economic journals. This is already achieved by our team of collaborators for studies with publicly available data. We also provide assistance to data editors to reproduce studies using non-public data (e.g., proprietary data) through our collaborations with researchers and research institutes having access to such data. We also plan to help the data editor of the Canadian Journal of Economics. As of now, the data editor (Marie Connolly, a collaborator) only verifies the completeness of the replication packages, but does not reproduce the results. Lastly, our medium-term objective involves reproducing all studies with publicly available data for four additional journals without a data editor. Furthermore, with the help of one of the co-PIs’ team we are currently developing educational material and guidelines for replicators. We are also working on improvements to the Social Science Reproduction Platform (https://www.socialsciencereproduction.org/) using another funding source.
This project would start in May 2023 and end in April 2027.