Python for Reproducible Research

Four per cent of published notebooks reproduce. The path from a notebook that works on your machine to a repository someone else can run — in eleven files

Author

Behzad Samadi

Published

September 12, 2026

In 2019 a team at NYU took 1,159,166 Jupyter notebooks from 264,023 GitHub repositories and tried to run them.

24.11% executed without a single error. 4.03% produced the same results the author had recorded in them.

That is not a study of beginners. Those are public notebooks, on GitHub, from people who cared enough to share their work. And the obvious objection — the environments were missing, of course they failed — was pre-empted by the same authors two years later, when they re-ran the experiment inside Docker containers stuffed with every installable dependency and forced top-down execution order. Their own summary: “We achieved a reproducibility rate that ranged from 4.90% to 15.04%.” Fifteen per cent is the best case, with the environment problem solved for you.

This article is about the other eighty-five per cent. Not the tooling — its companion piece covers environments and packages — but the gap between an analysis that ran once and a repository someone else can clone.

This is a method, not a standard

Nothing here is a journal requirement or an institutional policy, and where funder and publisher rules are concerned I have deliberately stayed quiet rather than print a deadline I could not verify. Check your own funder, your own journal, and your own institution’s data office. What follows is what makes work runnable by a stranger, which is a lower bar than compliance and a more useful one.

What actually breaks

The failure causes are worth reading, because they are unglamorous and every one of them is fixable:

Cause	Share of failures
`ImportError` / `ModuleNotFoundError`	29.23%
`NameError`	14.53%
`FileNotFoundError` / `IOError`	12.59%

The first is the environment, and it is the one the tooling world has spent 2024–2026 solving. The other two are not environment problems at all.

NameError is hidden state. A notebook’s output shows what happened, not what would happen. You defined a variable in a cell you later edited, ran cells out of order, and the notebook now depends on a sequence that no longer exists anywhere. In the same corpus, 36.36% of notebooks with an unambiguous execution order had cells that ran out of order, and 76.90% had gaps in the execution counter.

FileNotFoundError is a path on your machine. /Users/you/Desktop/data.csv is not a dependency you can declare.

And two numbers from that study reframe the whole problem:

10.30% of valid Python notebooks import any local module.
1.54% import a test framework.

Nine notebooks in ten contain every line of their logic inline, and ninety-eight in a hundred have nothing checking that logic is right. That is not a tooling gap. That is the notebook being used as the whole project rather than as the front end of one.

The fair hearing

It would be easy to conclude that notebooks are the problem. The strongest argument against that comes from the people who built them.

Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks (PLOS Computational Biology, 2019) has Fernando Pérez — who created IPython — among its authors. It makes the positive case, that notebooks “combine code, results, and descriptive text in a single ‘computational narrative’”, and concedes the failure mode in the same breath: they “can delete key steps or introduce ‘hidden state’ that confounds analyses and confuses readers”.

That is the right posture. The notebook is an excellent place to think and a bad place to keep the only copy of your logic. Joel Grus’s well-known 2018 talk was titled I Don’t Like Notebooks; his follow-up was titled Reproducibility: A Trojan Horse for Software Engineering Best Practices — which is this article’s argument in a subtitle. Nobody wants to write tests. People do want their results to survive a reviewer.

Organize: get the logic out of the notebook

The move is not “stop using notebooks”. It is to demote the notebook from the project to the narrative, and put the logic somewhere it can be imported and tested.

Concretely: the notebook keeps the story, the figures, the prose and the judgement calls. A module keeps anything you would be upset to lose, anything you call twice, and anything you could be wrong about.

The `sys.path` trap

The obvious way to import your own code into a notebook is the wrong one:

import sys
sys.path.append("../src")   # don't
from mymodule import clean

This works until the notebook is opened from a different directory, or run by someone else, or run by CI — at which point the relative path is wrong and you are back to ModuleNotFoundError, which we have already established is the single largest cause of failure.

The fix is to make your project a real, installed package, which takes four lines:

# pyproject.toml
[project]
name = "bocrates"
version = "0.1.0"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Install it in editable mode once, and from bocrates import annual_average works from the notebook, from the test suite, from a script and from CI, with no path manipulation anywhere. Editable means your edits take effect immediately — you are not copying code into an environment, you are telling the environment where your code lives.

`src/` or not

There is a long-running argument about whether your package should sit at the repository root or under src/. The case for src/ is narrow and real: if your code is at the root, import mymodule may succeed from the project directory even when the package is not installed, because the current directory is on the path. Your tests then pass locally and fail for everyone else. A src/ layout makes that impossible — if it imports, it is installed.

For an analysis with a test suite, that is worth the extra directory. (Note if you go looking: Cookiecutter Data Science removed its src/ directory in v2, renaming it to the project module, so older tutorials describe a layout its current version no longer has.)

Test one thing

The 1.54% figure is the most damning number in this article, and the usual advice — “write unit tests” — does not fix it, because it asks a researcher to specify correct behaviour they may not be able to state.

So do not start there. Start with a characterisation test: a test that does not claim today’s answer is right, only that it has not silently changed.

You already have the precondition. You trust today’s output — that is why you put it in the paper. Freeze it:

def test_output_matches_baseline():
    result = run_analysis(DATA)
    expected = np.load("tests/baseline.npy")
    np.testing.assert_allclose(result, expected, rtol=1e-6)

Ten lines. It catches the refactor that changed a boundary, the dependency upgrade that changed a default, the off-by-one you introduce next March. It does not catch “you were wrong all along” — nothing catches that but thinking — and it is still the highest-value test in a research repository, because the thing most likely to break your analysis is you, later.

Two numerical traps worth knowing

pytest.approx and assert_allclose disagree about “close”. approx defaults to a relative tolerance of 1e-6 with an absolute floor of 1e-12; assert_allclose defaults to 1e-7 relative with atol=0, which gives no free pass near zero. Comparing anything that should be zero will behave differently under the two. Pick deliberately.

Seeding is not as portable as it looks. NumPy’s legacy RandomState API carries a documented stream-compatibility guarantee; the modern default_rng is the better generator and does not promise the identical bit stream across versions. If your test asserts exact values from random draws, you have coupled your test suite to a NumPy version. Assert on statistics, or pin hard and know why.

Make it citable

The last stage is the one most analyses skip, and it costs about ten minutes.

Archive it. Connect the repository to Zenodo, cut a GitHub release, and Zenodo mints a DOI for that release. You get two: a version DOI for the exact release, and a concept DOI that always resolves to the latest. Cite the version DOI in a paper — the whole point is that it cannot change under the reader.

Tell people how to cite it. Add a CITATION.cff — plain text, and the current schema version is 1.2.0:

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
title: "bocrates: annual summaries of the Bank of Canada overnight rate"
version: 0.1.0
date-released: 2026-09-12
authors:
  - family-names: Samadi
    given-names: Behzad

GitHub renders it as a “Cite this repository” button in APA and BibTeX. And Zenodo reads it when you release — “Zenodo will use the citation information you’ve provided to populate the publication entry” — so it is worth adding before the first release rather than after.

Link rot is a reproducibility failure

The Turing Way’s own project template currently links to a page on its old domain that no longer resolves. This is not a criticism of a genuinely excellent project — it is the most honest possible demonstration that a URL is not an archive. A DOI is a promise about the future; a link is a hope about it.

The whole thing, in eleven files

None of this is hypothetical. The example below is committed, runnable, and was run before this article was written.

example/
├── data/overnight.csv          69 monthly observations, Bank of Canada
├── src/bocrates/
│   ├── __init__.py
│   └── summary.py              the logic: pure functions, no paths
├── tests/test_summary.py       5 tests
├── analysis/annual_summary.py  regenerates every output
├── outputs/                    NOT committed — regenerable by definition
├── pixi.toml  pixi.lock        the environment, and exactly what it resolved to
├── pyproject.toml              makes src/ importable without sys.path games
├── CITATION.cff                how to cite it
├── README.md                   how to run it
└── .gitignore                  outputs, caches, environments

The logic is a pure function — everything arrives as an argument, everything leaves as a return value:

def annual_average(observations: list[tuple[str, float]]) -> dict[int, float]:
    """Mean rate per calendar year, keyed by year."""
    buckets: dict[int, list[float]] = defaultdict(list)
    for date, rate in observations:
        buckets[int(date[:4])].append(rate)
    return {year: sum(rates) / len(rates) for year, rates in sorted(buckets.items())}

No file paths, no printing, no globals. That is not stylistic fastidiousness — it is precisely what makes the next four lines possible:

def test_averages_within_a_year():
    got = annual_average([("2021-01-01", 1.0), ("2021-07-01", 2.0)])
    assert got == {2021: pytest.approx(1.5)}

And the commands that reproduce the work are themselves recorded, in pixi.toml:

[tasks]
test    = { cmd = "pytest -q" }
summary = { cmd = "python analysis/annual_summary.py" }
all     = { depends-on = ["test", "summary"] }

So the whole contract is one line — pixi run all — which runs the tests and then regenerates outputs/annual-average.csv from data/overnight.csv. Running it twice produces byte-identical output.

One honest inconsistency

The companion article says that for pure Python, uv is the reasonable default — and this example is pure Python, with no native dependencies at all. It uses pixi anyway, for one reason: uv has no task runner. For an analysis, the environment is only half the reproducibility story; the commands you actually ran are the other half, and pixi run all records them where a README cannot enforce them.

That is a real trade-off rather than a clean answer, and you should make it deliberately. Pure-Python work with a Makefile or just file would be equally defensible.

What this does and does not buy you

A caution, carried over from the companion article because it matters more here.

The Turing Way distinguishes four things: a result is reproducible when the same analysis on the same data gives the same answer; replicable across different data; robust across different analyses; generalisable when both.

Everything in this article buys you the first one. That is the floor, not the ceiling — and it is worth being blunt about what the floor is worth: a perfectly reproducible pipeline guarantees that your wrong answer is wrong identically forever. It makes error findable, which is the actual argument. Nobody can check work they cannot run.

A vocabulary trap

The Turing Way and ACM’s artifact-badging vocabulary use “reproduced” and “replicated” in opposite senses. If you are writing for a venue that badges artifacts, check which definition applies before you use either word in a claim.

What to do this week

Move one function out of your notebook into a module, and import it back. Not all of them. One.
Add a pyproject.toml and install your project editable, so the import works everywhere and you never write sys.path.append again.
Write one characterisation test that freezes today’s output. You do not need to know the right answer to do this.
Put the commands in a task file so “how do I run it” has an answer that is not a paragraph in a README.
Add a CITATION.cff before your next release, so the archive has your metadata rather than guessing.

The test of whether it worked is not a checklist. It is the one from the course this article grew out of: hand the repository to someone else, and see whether they can regenerate your results without asking you a question.

Sources

Research notes, including what could not be verified and the numbers I chose not to print, are in the accompanying folder. The worked example is docs/2026-09-12-python-reproducibility/example/.

The notebook study. Pimentel, Murta, Braganholo & Freire, A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks, MSR 2019, doi:10.1109/MSR.2019.00077; and the authors’ own tougher replication, Understanding and improving the quality and reproducibility of Jupyter notebooks, Empirical Software Engineering, 2021, doi:10.1007/s10664-021-09961-9.
The replication on published science. Samuel & Mietchen, Computational reproducibility of Jupyter notebooks from biomedical publications, GigaScience, 2024, doi:10.1093/gigascience/giad113.
The defence, from the notebook’s own authors. Rule, Birmingham, Zuniga, Altintas, Huang, Knight, Moshiri, Nguyen, Rosenthal, Pérez & Rose, Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks, PLOS Computational Biology, 2019, doi:10.1371/journal.pcbi.1007007.
Layout. PyPA on src layout vs flat layout.
Size limits. GitHub docs on large files on GitHub.
Citation and archiving. Citation File Format (schema 1.2.0); Zenodo.
Definitions. The Turing Way.