Diagnostics lab
Part 2½ of a tutorial on computationally reproducible research. This lab assumes the failure-mode taxonomy.
The scenario
A (fictional) paper reports an attention study:
Participants high in trait anxiety responded more slowly than participants low in trait anxiety, mean difference = 65.9 ms, 95% bootstrap CI [48.0, 84.0].
You have the authors’ data and their analysis scripts, reproduced below as four steps sharing one workspace, exactly as they’d share a notebook. Your job: run the pipeline, watch it fail to reproduce the paper, diagnose why, and fix it until the final check passes. There are three distinct reproducibility bugs, one per analysis step, each an instance of a failure mode from Part 2.
Work top to bottom. After any edit, re-run the edited cell and everything
below it (you know why: implicit
state). If a later
step reports a NameError, an earlier step hasn’t run yet in this session;
go back and run the steps in order.
Step 1 · Load the data
Hint (step 1)
You are not Ana. Where does the file actually live? The data file has been placed in the working directory for you, the same place a well-organized repository would keep it, relative to the project root.
Answer (step 1)
Bug: implicit state, in the form of a hardcoded absolute
path. Change the path to the relative "rt_data.csv"
and re-run: 24 rows. In a real project: all paths relative to the project
root, and the README says where to put (or how to fetch) the data.
Step 2 · Apply inclusion criteria
The authors excluded two participants who failed attention checks (P23, P24), then, per their preregistration, analyzed “the first 20 of the remaining participants.”
Re-running the cell above gives the same 20 every time, which is exactly what makes this bug treacherous: hash randomization is fixed per interpreter process, so on your machine the selection looks stable. The question is whether Ana’s machine picked the same 20. The cell below runs the same selection in a throwaway fresh interpreter each time, as if on a different computer (your lab workspace above is untouched). Run it two or three times and watch who gets excluded.
Hint (step 2)
What determines the order of list(passed)? (You saw this in
failure
mode 1.) Note that the lab cell's printout is sorted for display, which
is exactly how this bug hides in real logs.
Answer (step 2)
Bug: hidden randomness, via "first 20" of a
set. A set's iteration order depends on per-process
hash randomization, so list(passed)[:20] selects a different
subsample on (almost) every machine. No error, plausible output, and a
different wrong answer everywhere. Fix: make the ordering explicit:
included = sorted(passed)[:20], which selects P01–P20, matching
the paper. Then re-run steps 3 and 4.
Step 3 · Estimate the effect
Hint (step 3)
Run this cell twice without changing anything. Does the CI hold still? Should it?
Answer (step 3)
Bug: hidden randomness, via an unseeded bootstrap. The
point estimate is deterministic, but the resampling isn't: every run gives
a slightly different CI, none of them the paper's. Fix: seed the generator
before resampling by adding random.seed(2026) on the line
before lo, hi = boot_ci(high, low) (the seed the authors
reported). The CI is now bit-stable run to run: bit-exact given the seed,
statistically stable across seeds, which is failure mode 6's two-layer
standard.
Step 4 · Check against the paper
This cell is the referee: it compares your pipeline’s numbers against the published claim.
Full solution (all three bugs)
- Step 1, implicit state: hardcoded absolute path →
path = "rt_data.csv". - Step 2, hidden randomness: set-ordered inclusion →
included = sorted(passed)[:20]. - Step 3, hidden randomness: unseeded bootstrap →
random.seed(2026)immediately beforeboot_ci.
With all three fixes, re-run steps 1→4 in order: difference 65.9 ms, CI [48.0, 84.0], and step 4 prints REPRODUCED.
What to take away
Each bug leaves behind a habit:
- Paths: if an absolute path appears anywhere in analysis code, it is a reproduction failure waiting for its audience.
- Order: any time your code depends on the order of a collection, ask who guarantees that order. If the answer is “nobody,” sort.
- Seeds: the moment randomness enters, the seed is part of the method. Report it like you’d report the sample size.
And one meta-lesson: notice how the pipeline never raised an error. Every bug produced plausible numbers. The check in step 4, an executable statement of what the paper claims, is what made the failures visible, and it’s a miniature of what continuous verification looks like in Part 3.
Discussion questions
- The three bugs raised no errors. What is your closest near-miss with a silently wrong number, and what caught it?
- If your current project had a step-4-style check, what three lines would
be in its
CLAIMStable? - In your lab, whose job is it to catch bugs like these before publication? Is that assignment real or notional?
Next: Part 3 · Interventions.