Defintions


A variety of definitions are used for the terms reproducibility and replicability. See, for instance, Bollen et al. (2015) and a 2019 report from the National Academies of Sciences, Engineering, and Medicine. We adopt the following definitions:

Computational Reproducibility: The ability to duplicate the results of a prior study using the same data and procedures as were used by the original investigator. Reproducibility is done using the same computer code (possibly rebuilt from scratch), but can be achieved using a different software package.

Robustness Replicability: The ability to duplicate the results of a prior study using the same data but different procedures as were used by the original investigator. Robustness replicability can be done using the raw, intermediate or final data sets used by the original authors.

Direct Replicability: The ability to duplicate the results of a prior study using new data but the same procedures as were used by the original investigator.

Conceptual Replicability: The ability to duplicate the results of a prior study using new data and different procedures as were used by the original investigator.


Bollen, K., Cacioppo, J. T., Kaplan, R. M., Krosnick, J. A., & Olds, J. L. (2015). Social, behavioral, and economic sciences perspectives on robust and reliable science [Report of the Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences]. National Science Foundation. https://www.nsf.gov/sbe/AC_Materials/SBE_Robust_and_Reliable_Research_Report.pdf
National Academies of Sciences, Engineering, and Medicine. (2019). Methods to Foster Transparency and Reproducibility of Federal Statistics: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi.org/10.17226/25305
National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and replicability in science. National Academies Press. doi.org/10.17226/25303