---
title: "Reports That Can't Disagree With Their Data"
pubDate: 2026-11-18T00:00:00.000Z
tags:
  - Software Engineering
  - Data Engineering
  - Agentic Coding
image: /images/blog/reports-that-cant-disagree/featured.png
---

import PricingFigure from '../../components/PricingFigure.astro';

## TL;DR
In an analysis report, the prose is where numbers go stale: a sentence still says 4.5% long after a rerun made it 4.4%. The motor-pricing analysis behind three earlier posts, on [the data](/post/fremtpl2-head-block), [large claims](/post/large-claim-loading) and [the model comparison](/post/tweedie-glm-vs-xgboost), is built so that can't happen. A pipeline computes everything; notebooks may only read its results; a check fails the render on any number typed into the prose; and renders are deterministic, so every change to the committed reports is a real one. Most of the code was written with a coding agent, and these checks are what make its edits safe to accept.

## The pipeline computes, the notebooks present

Every computation is a rule in a [Snakemake](https://snakemake.github.io/) workflow, from the download through cleaning, the split, one job per model and fold, the benchmark and the final model. Rules write layered data (raw, cleaned, model outputs) and, per topic, a `stats.json` and a few small tables: the only thing a report may read. All parameters, from cleaning caps to seeds, live in one config file, and Snakemake reruns a rule exactly when its inputs, parameters or code change. Changing one model's learning rate refits that model's five folds and nothing else.

<PricingFigure name="pipeline" />

*Rules write result files; notebooks only read them. Checks guard the way to the reports.*

A Quarto notebook reads through two functions, `read_stats(topic)` and `read_table(topic, name)`, then reshapes, plots and explains. Each notebook's render rule finds its inputs by scanning the notebook for the topics it reads, so a report re-renders exactly when a result it shows changes, and a render takes seconds.

## The checks

**No typed numbers.** Every number in the prose is an inline expression, evaluated at render time:

```markdown
It explains `{python} f"{d2[best]:.2%}"` of the constant model's deviance,
the Tweedie GLM `{python} f"{d2['glm_tweedie']:.2%}"`.
```

A pre-render script blanks out everything that isn't prose (code, inline expressions, math, link targets), removes a short whitelist such as years and identifiers like `log1p`, and fails the render on any digit left. Captions and Markdown tables are covered too, so they come from code. Claims about numbers are computed as well: "beats the GLM by far" is chosen from the results, so a rerun that changes the comparison changes the sentence.

**Notebooks may not compute.** A second script fails the render if a code cell reads layer data, writes a file, calls `.fit(`, imports a modelling library or calls a pipeline computation. A new number has to come from a rule, computed once with the same folds and seeds as everything else.

**Renders are deterministic.** Committed reports are only reviewable if a rerun that changes nothing changes nothing in the diff. Plotly gives every figure a random div ID, so a wrapper sets IDs from a counter; matplotlib's SVGs hash float clip boxes into their IDs, so its few figures are PNGs; tables get fixed IDs. A pre-push hook then re-renders every report from the existing results and refuses the push if the committed reports differ. CI runs the static checks: ruff with every rule on, `mypy --strict`, both notebook checks and a dry run of the pipeline.

## What the checks caught

Computed prose doesn't make a number right, only traceable. In the data-size study, the GLM's mean $D^2$ on 10,000 rows rendered as a 43-digit negative percentage. The sentence around it was computed, so the absurd value went straight into the report rather than being smoothed over by hand, and the cause was easy to trace: one fold's GLM extrapolated its log-linear bonus-malus tail from a handful of policies, and its premiums overflowed. The fix holds the tail constant above the largest value seen in fitting, as the splines already were.

## A model too slow for CI

TabICL, the tabular foundation model in the comparison, takes minutes on a laptop GPU and would take hours on CI's two CPU cores. So its out-of-fold predictions are committed with a fingerprint: a hash of the training rows it sees (rounded, so last-digit float differences between machines don't count), its parameters and the source of its wrapper. Without TabICL installed, the pipeline reuses the frozen predictions only while the fingerprint matches, and otherwise stops and asks for a local rerun. XGBoost's results still differ in the third digit between macOS and Linux, so all committed reports come from one machine, and the reports say so.

## Why this matters with a coding agent

The conventions live in an `AGENTS.md`, written for coding agents and humans alike, and most commits were made together with one. An agent writes a paragraph around a figure quickly, and just as quickly writes into it a number it remembers from an earlier run. Here that fails the render. Asked for a new statistic in a notebook, it can't compute it there; it has to add it to a rule. Reviewing its change means reading a diff of code and a diff of rendered reports, and the second shows only what really changed.

[The Missing Test Suite](/post/the-missing-test-suite) argues that once code is cheap, the bottleneck is knowing whether the output is right. For an analysis, part of that can be made mechanical: not whether the method is sound, which still takes someone who knows the field, but whether the report says what the data say.
