Part 1 of a tutorial on computationally reproducible research. Next: failure modes.

Two questions that sound alike

Take any published quantitative finding. There are two very different ways to ask whether it holds up:

1. If I rerun the analysis, with the same data and the same code, do I get the same numbers?
2. If I redo the study, with new data and the same question, do I reach the same conclusion?

The first is reproducibility; the second is replicability. The National Academies’ consensus report (2019), the closest thing the field has to a standard, defines them this way:

Reproducibility is obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis. … Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. National Academies of Sciences, Engineering, and Medicine (2019)

The report adds, pointedly: “As defined by the committee, reproducibility relates strictly to computational reproducibility.” That is this tutorial’s subject. Replication, with all its questions of theory, measurement, and sampling, is a different and larger topic; reproducibility is the humbler precondition. If the same data and code don’t yield the same numbers, there is nothing stable to replicate.

The Turing Way arranges the possibilities in a 2×2 that is worth committing to memory:

The third cell, robustness (same data, different defensible analysis choices), will come up repeatedly, because it turns out results often depend on those choices far more than anyone expects.

The boundary is genuinely fuzzy

If you find the terminology confusing, you are in excellent company; the confusion is structural, not personal. Barba (2018) surveyed usage across fields and found three incompatible camps: fields that use the words interchangeably; fields that use reproduce for same-data, same-code (the convention here, inherited from Claerbout and Peng); and fields, including parts of computer science, microbiology, and FASEB, that define them exactly backwards relative to the second group. As she put it, “different groups are using terminologies in utter contradiction with each other.”

The best single illustration: the ACM, which awards “Results Reproduced” badges on papers, used the reversed definitions from 2016 to 2020, then swapped the meanings of the two words in August 2020 to align with the National Academies. A badge on a 2019 paper and the same badge on a 2021 paper certify different things. So when you read any claim about “reproducibility rates,” check which definition is in force before you quote the number.

The lineage of an idea

Computational reproducibility as a discipline has a birthplace: exploration geophysics. Jon Claerbout’s Stanford group, drowning in un-rerunnable seismic analyses, rebuilt its research process in the early 1990s so that every figure in a dissertation could be regenerated by a single command (Claerbout & Karrenbach, 1992). Buckheit and Donoho, importing the practice to statistics with WaveLab, distilled the philosophy into the lines quoted on the main page: the article is merely advertising of the scholarship; the actual scholarship is the environment and the instructions that generated the figures.

Peng (2011) turned this into a practical spectrum. Between “not reproducible” (publication only) and the gold standard (full replication) lie graduated states: publication plus code; plus code and data; plus linked and executable code and data. Two ideas in that framing do a lot of work:

• Reproducibility is a spectrum rather than a binary. Projects move along it one artifact at a time, and every step counts.
• Reproducibility is “a minimum standard for judging scientific claims when full independent replication of a study is not possible.” It is a floor rather than a ceiling.

Goodman, Fanelli, and Ioannidis (2016) offer a finer-grained lexicon if you need one: methods reproducibility (can the procedures be re-executed exactly?), results reproducibility (does a new study corroborate?), and inferential reproducibility (do the same data support the same conclusions?). For this tutorial, the National Academies definitions suffice.

How close is “the same”?

Here is a question that sounds pedantic and is actually foundational: if you rerun an analysis and get 0.4999997 where the paper says 0.50, did it reproduce?

There are three defensible standards, and mature projects choose deliberately among them:

• Bit-exact. The strictest standard: identical bytes out. Hinsen’s argument for it is epistemic: “If two computations lead to similar but slightly different results, how do you decide if this is due to a bug or to some ‘inevitable’ fuzziness? … you can’t.” Bit-exactness makes verification trivial (diff suffices), and it is achievable far more often than people assume.
• Bounded-error / statistical. Results match within a stated tolerance, or estimated quantities agree within Monte Carlo error. This is the right standard for stochastic computation (MCMC, bootstrap, stochastic optimization) when the stochasticity itself can’t or shouldn’t be pinned, and for floating-point work across platforms. The National Academies explicitly allow results within “an accepted range of variation.”
• Semantic. The conclusions (sign, significance, approximate magnitude) survive even if the digits differ. This is the weakest standard, and the one most audits fall back to implicitly. Say so explicitly if it’s the one you’re using.

Large-scale efforts operationalize this concretely: the reproducibility arm of a major program in the social and behavioral sciences (Miske et al., 2026) scored a claim as precisely reproduced when the recomputed numbers matched, and approximately reproduced when effect sizes landed within 15% of the original and p-values within 0.05. Under those definitions, 53.6% of claims reproduced precisely and 73.5% approximately. The gap between those two numbers is the tolerance question, quantified.

A useful discipline you’ll meet again in the audit: decide for each result which standard applies. A table of descriptive statistics should be bit-exact. A bootstrapped confidence interval should be bit-exact given the seed, and statistically stable across seeds. A conclusion should survive all of it.

By the way, this page can run Python. Here’s the tolerance question in its smallest possible form (run it, then try == in place of isclose):

What the evidence shows

How often does published research clear even the low bar? The reproduction literature now spans a decade and several hundred audited papers. Three patterns recur.

Pattern 1: availability is the first wall. Before anyone can rerun anything, the materials must exist somewhere findable.

Pattern 2: even with materials in hand, exactness is roughly a coin flip.

That last row deserves a second look: it is as close as the literature gets to a controlled experiment on transparency, and sharing code roughly doubled the exact-reproduction rate (48% to 77%).

Pattern 3: analytic flexibility is its own, independent source of variation. Even perfect reproduction of one pipeline understates the uncertainty, because defensible pipelines disagree:

Why failures stay invisible

A final foundation before the taxonomy: reproducibility failures are distinctive in that nothing announces them. Code that depends on an unseeded RNG, an unpinned library, or an undocumented manual step runs without complaint today, on your machine. The failure only comes into view when someone attempts reproduction, which for most papers is never (and, when it does happen, is usually years later, when the laptop is gone and the postdoc has moved on). The Hardwicke et al. authors captured the implied standard drily: “repeating the original analyses upon the original data should yield the original outcomes.” The surprise is only that this needs saying.

This is why the next part of the tutorial is organized the way it is. You cannot fix what you cannot see, so the first skill is diagnosis: knowing the small set of mechanisms by which working code quietly becomes irreproducible code.

Next: Part 2 · Failure modes, where each of those mechanisms fails live, in your browser.