led by Ryan Garver · Boise, Idaho Get Smart AI, led by Ryan Garver
Email Ryan Hiring?
Back to work

Design + build · 2026 · public · 74 tests · reproducible

model-regress

An eval harness that tells you whether a model swap is a real regression or just the model disagreeing with itself, with the statistics to back the call.

Source
75 pts pass-rate gap, measured Sonnet 5 (38/40) vs Haiku 4.5 (8/40) on a frozen 40-item golden set, Sep 2026; paired 95% CI 57.5 to 85.8, against a same-model repeat disagreement of 4/40
$0.24 run API spend the paired run; a separate $0.12 pilot that surfaced an underspecified prompt is disclosed and withdrawn, not hidden
~20 pts min detectable effect the published power analysis: at n=40, catching a 5-point regression would need about 628 items, and the protocol says so out loud 
$ model-regress estimate --all
A (claude-sonnet-5):  expected $0.0869, 40 calls
B (claude-haiku-4-5): expected $0.0434, 40 calls
A-repeat (sonnet-5):  expected $0.0869, 40 calls  # noise ceiling

$ model-regress power --n 40 --discordant 0.2
MDE at n=40, power 0.8: 19.81 pp  (5 pp -> need n=628)
Cost and statistical power computed before any model is called. The A-vs-A-repeat run measures how much the model disagrees with itself, so a real regression can be told apart from run-to-run noise.

When a team swaps models or edits a prompt, how do they know whether a pass-rate change is a real regression or just the model disagreeing with itself run to run? Most comparisons run each condition once and report the delta as if it were noise-free. This harness measures the noise first, then decides against it.

What it found

On a frozen 40-item extraction set (September 2026), Claude Sonnet 5 passed 38 of 40 and Claude Haiku 4.5 passed 8 of 40: a 75-point paired difference (95% CI 57.5 to 85.8), against a same-model repeat disagreement of only 4 of 40. That is a real regression, not noise. Cost per item was $0.0026 versus $0.00076, and the whole run cost $0.24 of real API calls.

How it separates signal from noise

It runs three conditions over one frozen golden set: the current model, the candidate model, and the current model a second time. That third run, the model against itself on identical input, is the noise ceiling, measured in the same units as the effect. The decision rule was frozen before any data was collected: call it a regression only if the paired-difference confidence interval excludes zero and the effect exceeds the model's own run-to-run disagreement. A cost increase past a set threshold fails the gate regardless of pass rate, so a quality win cannot quietly ship a cost regression.

What it will not claim

The write-up publishes a power analysis: at this sample size the smallest reliably detectable effect is about 20 points, and catching a 5-point regression would need roughly 628 items, not 40. So the tool reports "no detectable regression at n=40" rather than a false "no regression" whenever an effect sits below what the sample can see. A $0.12 pilot run that surfaced an underspecified prompt is disclosed and withdrawn, dated before the real run, rather than buried.

How it is checked

74 tests pass and the type-check is clean. The result sentence is rendered from the committed run data, never hand-typed, and a reproduce command re-derives it bit for bit offline. The source, protocol, and data are public: read it and run it yourself.