Skip to main content
Software Engineering Data Engineering Agentic Coding
View as Markdown Suggest changes

Reports That Can't Disagree With Their Data

· Reading time: 5 min
Reports That Can't Disagree With Their Data

TL;DR

In an analysis report, the prose is where numbers go stale: a sentence still says 4.5% long after a rerun made it 4.4%. The motor-pricing analysis behind three earlier posts, on the data, large claims and the model comparison, is built so that can’t happen. A pipeline computes everything; notebooks may only read its results; a check fails the render on any number typed into the prose; and renders are deterministic, so every change to the committed reports is a real one. Most of the code was written with a coding agent, and these checks are what make its edits safe to accept.

The pipeline computes, the notebooks present

Every computation is a rule in a Snakemake workflow, from the download through cleaning, the split, one job per model and fold, the benchmark and the final model. Rules write layered data (raw, cleaned, model outputs) and, per topic, a stats.json and a few small tables: the only thing a report may read. All parameters, from cleaning caps to seeds, live in one config file, and Snakemake reruns a rule exactly when its inputs, parameters or code change. Changing one model’s learning rate refits that model’s five folds and nothing else.

Configcaps, seed,cap quantile,hyperparametersRulesdownloadeda · silvergold · fit_foldbenchmarkfinal_modelResultsstats.jsonsmall tablesper topicNotebooksread resultsdraw, explaincompute nothingReportsHTML, PDF,pipeline graph,in gitread-only checkno typed numberspre-push render

Rules write result files; notebooks only read them. Checks guard the way to the reports.

A Quarto notebook reads through two functions, read_stats(topic) and read_table(topic, name), then reshapes, plots and explains. Each notebook’s render rule finds its inputs by scanning the notebook for the topics it reads, so a report re-renders exactly when a result it shows changes, and a render takes seconds.

The checks

No typed numbers. Every number in the prose is an inline expression, evaluated at render time:

It explains `{python} f"{d2[best]:.2%}"` of the constant model's deviance,
the Tweedie GLM `{python} f"{d2['glm_tweedie']:.2%}"`.

A pre-render script blanks out everything that isn’t prose (code, inline expressions, math, link targets), removes a short whitelist such as years and identifiers like log1p, and fails the render on any digit left. Captions and Markdown tables are covered too, so they come from code. Claims about numbers are computed as well: “beats the GLM by far” is chosen from the results, so a rerun that changes the comparison changes the sentence.

Notebooks may not compute. A second script fails the render if a code cell reads layer data, writes a file, calls .fit(, imports a modelling library or calls a pipeline computation. A new number has to come from a rule, computed once with the same folds and seeds as everything else.

Renders are deterministic. Committed reports are only reviewable if a rerun that changes nothing changes nothing in the diff. Plotly gives every figure a random div ID, so a wrapper sets IDs from a counter; matplotlib’s SVGs hash float clip boxes into their IDs, so its few figures are PNGs; tables get fixed IDs. A pre-push hook then re-renders every report from the existing results and refuses the push if the committed reports differ. CI runs the static checks: ruff with every rule on, mypy --strict, both notebook checks and a dry run of the pipeline.

What the checks caught

Computed prose doesn’t make a number right, only traceable. In the data-size study, the GLM’s mean D2D^2 on 10,000 rows rendered as a 43-digit negative percentage. The sentence around it was computed, so the absurd value went straight into the report rather than being smoothed over by hand, and the cause was easy to trace: one fold’s GLM extrapolated its log-linear bonus-malus tail from a handful of policies, and its premiums overflowed. The fix holds the tail constant above the largest value seen in fitting, as the splines already were.

A model too slow for CI

TabICL, the tabular foundation model in the comparison, takes minutes on a laptop GPU and would take hours on CI’s two CPU cores. So its out-of-fold predictions are committed with a fingerprint: a hash of the training rows it sees (rounded, so last-digit float differences between machines don’t count), its parameters and the source of its wrapper. Without TabICL installed, the pipeline reuses the frozen predictions only while the fingerprint matches, and otherwise stops and asks for a local rerun. XGBoost’s results still differ in the third digit between macOS and Linux, so all committed reports come from one machine, and the reports say so.

Why this matters with a coding agent

The conventions live in an AGENTS.md, written for coding agents and humans alike, and most commits were made together with one. An agent writes a paragraph around a figure quickly, and just as quickly writes into it a number it remembers from an earlier run. Here that fails the render. Asked for a new statistic in a notebook, it can’t compute it there; it has to add it to a rule. Reviewing its change means reading a diff of code and a diff of rendered reports, and the second shows only what really changed.

The Missing Test Suite argues that once code is cheap, the bottleneck is knowing whether the output is right. For an analysis, part of that can be made mechanical: not whether the method is sound, which still takes someone who knows the field, but whether the report says what the data say.

AI Chat

Messages you send are processed by the Google Gemini API to generate responses. Do not share sensitive personal data. See the privacy policy for details.