Companion to a tutorial on computationally reproducible research.

Start here

Five short pieces carry most of the field’s wisdom:

• Sandve, Nekrutenko, Taylor & Hovig (2013), “Ten Simple Rules for Reproducible Computational Research”: still the best single checklist; rules 3 (archive exact versions) and 6 (note the seeds) alone would fix half the failures in Part 2.
• Wilson et al. (2017), “Good enough practices in scientific computing”: the minimal-fix philosophy, expanded into a full practice guide, explicitly designed for researchers without software engineering training.
• Peng (2011), “Reproducible research in computational science”: the reproducibility spectrum in two pages.
• Barba (2018), “Terminologies for reproducible research”: read before quoting anyone else’s definitions.
• Hinsen (2019), “Dealing with software collapse”: why working code stops working, and what “maintenance” actually means for research software.

Handbooks and courses

• The Turing Way: the community-written handbook of reproducible, ethical, collaborative research; the source of the 2×2 definitions matrix in Part 1 and a good next stop on nearly any topic here.
• The Carpentries: workshops and open lessons on the underlying skills, including the Unix shell, Git, R for reproducible analysis, and automation with Make.
• CodeRefinery: free workshops one level up from Carpentries (testing, modular code, containers, reproducible workflows); materials all open.
• The Practice of Reproducible Research (Kitzes, Turek & Deniz, eds., 2018): 31 case studies of real projects, including the compromises.
• Project TIER: a prescribed folder-and-script protocol for empirical social science; opinionated in a useful way for teaching.
• Christensen, Freese & Miguel (2019), Transparent and Reproducible Social Science Research: the textbook treatment, from the BITSS orbit.
• Building Reproducible Analytical Pipelines with R (Rodrigues): renv, targets, Docker, and Nix for R users, end to end.

Communities and events

• ReproducibiliTea: grassroots journal-club network (130+ institutions, 30+ countries); the lowest-cost way to build local culture.
• ReproHack: hackathons where participants try to reproduce volunteered papers and send authors structured feedback; the event-shaped version of this tutorial’s lab.
• Society for the Improvement of Psychological Science (SIPS) and FORRT: meta-science and open-scholarship training communities; FORRT’s glossary is the reference for terminology beyond Part 1.
• CODECHECK: independent executions of papers’ code, certified with a time-stamped report; both a service to use and a model to imitate.

The evidence: reproduction and robustness studies

The full citations behind the tutorial’s numbers, for teaching or quoting:

• Stodden, Seiler & Ma (2018), PNAS: 204 Science articles; artifacts for 44%, reproduced 26%.
• Hardwicke et al. (2018) and (2021), Royal Society Open Science: analytic reproducibility at Cognition and Psychological Science; author assistance required in most successful cases.
• Obels et al. (2020), AMPPS: Registered Reports; 58% of shareable pipelines reproduced.
• Crüwell et al. (2023), Psychological Science: 1 of 14 badged articles exactly reproducible.
• Miske et al. (2026), Nature: the largest audit to date; 600 papers, 53.6% precise / 73.5% approximate among assessable claims; data availability 24%.
• Aczel et al. (2026), Nature: analytic robustness with ≥5 reanalysts per paper; 34% matched within ±0.05 Cohen’s d.
• Breznau et al. (2025), Royal Society Open Science: the transparency experiment; code sharing roughly doubles exact reproduction (48% to 77%).
• Many analysts, one dataset: Silberzahn et al. (2018) (soccer red cards); Botvinik-Nezer et al. (2020) (70 fMRI teams, no two identical workflows); Breznau et al. (2022) (1,253 models, >95% of variance unexplained); Gould et al. (2025) (ecology).
• Collberg & Proebsting (2016), CACM: repeatability in computer systems; the “does it even build?” baseline.
• Domain-specific failure modes: Bridges et al. (2020), the timing mega-study (experiment-software timing across platforms; measure your own rig); Pham et al. (2020) (variance across 2,304 identical deep-learning runs); Vehtari et al. (2021) (modern MCMC convergence standards: R̂ < 1.01, bulk/tail ESS).

Tools, by intervention

Environments. uv (Python; the current default), pixi (conda ecosystem, lockfile-first), Miniforge/conda-forge (if conda, then this flavor), renv (R), rig (pinning R itself).

Containers. Docker, Apptainer (HPC; formerly Singularity), Podman (rootless drop-in), Rocker (R images), Ten simple rules for Dockerfiles.

Workflows. Make, Snakemake, Nextflow, targets (R; successor to drake).

Literate/computational documents. Quarto, Jupyter Book, Binder (hosted executability: a bonus, never an archive).

Archiving & identifiers. Zenodo (DOIs; GitHub release integration), OSF, Software Heritage and SWHIDs (ISO-standard intrinsic software identifiers), RO-Crate (machine-readable research bundles), DataLad (versioned data on Git).

Tool advice rots faster than principles, so note where things stood as of 2026: prefer uv over pip/pipenv/poetry for new Python work; renv replaced packrat; targets replaced drake; Apptainer is Singularity’s current name; avoid the Anaconda defaults channel (licensing) in favor of conda-forge; and Papers with Code shut down in 2025 (Hugging Face hosts a successor). If you’re reading this in 2030: audit this paragraph first.

Reproducibility in the age of AI

Two fronts are moving fast, one hopeful and one cautionary.

AI as reproducer. Benchmarks now measure whether AI agents can reproduce papers’ results from their artifacts: CORE-Bench (agents went from ~21% to ~78% on its hardest tier within a year); PaperBench (from-scratch replication of ML papers; expert humans still ahead); SUPER; ReplicatorBench (agents for social-and-behavioral-science replication). The near-term implication: automated reproduction audits at scale are becoming feasible, which raises the value of every practice in this tutorial, because machine auditors are even less forgiving of missing lockfiles than graduate students.

AI as reproducibility hazard. Research that depends on hosted models inherits failure modes no lockfile can pin. Chen, Zaharia & Zou (2024) documented GPT-4’s behavior shifting substantially within months. Palmer, Smith & Spirling (2024) argue proprietary-model research requires explicit justification. Sclar et al. (2024) showed prompt formatting alone can swing benchmark accuracy by dozens of points. And temperature-0 sampling is not deterministic in practice; the root cause, batch-variant inference kernels, was dissected by Thinking Machines (2025). Reporting standards are emerging: REFORMS for ML-based science and a reporting checklist for LLMs in behavioral research. Minimal practice meanwhile: pin model snapshots, log every request and response, archive the transcripts as data, and treat “the model” as a versioned dependency that can be deprecated out from under you.

Foundations and history

• Claerbout & Karrenbach (1992): where “reproducible research” (and the single-command standard) began.
• Buckheit & Donoho (1995): WaveLab and the “advertising” credo.
• Schwab, Karrenbach & Claerbout (2000): the Stanford system, matured.
• Goodman, Fanelli & Ioannidis (2016): methods, results, and inferential reproducibility.
• National Academies (2019), Reproducibility and Replicability in Science: the consensus definitions and the policy landscape.
• ACM artifact review and badging: the badge taxonomy (and, historically, the great definition swap of 2020).

Back to the tutorial.