led by Ryan Garver · Boise, Idaho Get Smart AI, led by Ryan Garver
Email Ryan Hiring?
Back to work

Design + build · 2026 · public · 83 tests · reproducible

LLM Cost Autopilot

A cost auditor that prices real Claude traffic against the vendor rate table and reports, in dollars with a confidence interval, what prompt caching actually saved.

Source
84.7% bill saved by caching audited over 10,703 real Claude API calls, 85 of my own sessions, Aug to Sep 2026; the saving is Anthropic's prompt caching, priced and measured here (95% CI 83.2 to 86.0)
$3,959.81 actually billed against a $25,863.40 no-cache counterfactual: the same traffic with every input token at the fresh rate
10,703 real API calls priced 10,669 of them (99.7%) read from cache 
$ llm-cost headline
Across 10703 real Claude API calls in 85 of my own Claude
Code sessions (2026-08-16 to 2026-09-15), prompt caching cut
the bill from $25863.40 to $3959.81 actually billed: 84.7%
saved (session-level bootstrap 95% CI 83.2-86.0%); 10669 of
10703 calls (99.7%) read from cache.
The tool's own headline command against my Claude Code traffic (numbers verbatim, trimmed to fit). It refuses to print a number when no run exists, rather than fall back to a placeholder.

Every LLM bill mixes a few different things: what got billed at the full rate, what got a cache discount, and what could have been cheaper another way. Vendors publish the discount multipliers, but nothing turns that table into "here is what your own traffic actually saved." This tool does, in dollars, with a confidence interval instead of a single suspicious-looking number.

What it measured

Run against 10,703 real Claude API calls from 85 of my own Claude Code sessions (August to September 2026), it priced the traffic against a hand-verified rate table, then computed the counterfactual: the same calls with every input token billed at the fresh rate. Prompt caching cut the bill from $25,863.40 to $3,959.81 actually billed, a saving of 84.7% (session-level bootstrap 95% CI 83.2 to 86.0). 10,669 of the 10,703 calls read from cache.

The saving is Anthropic's prompt caching doing its job. What this tool adds is pricing and auditing that saving honestly, on real traffic, so the number is measured rather than promised.

Why the number is trustworthy

  • The percent-saved interval comes from a session-level cluster bootstrap, not a call-level one, because calls inside one session are not independent draws and resampling them individually would understate the interval.
  • Only caching goes in the headline. Two other savings, batching and cheaper-model routing, are computed but kept out: one needs a latency tolerance this traffic does not have on record, and the other assumes a cheaper model would have answered acceptably, which nothing here verifies. Routing is labelled quality-unverified in the tool's own output.
  • The headline command refuses to print anything when no run exists, rather than fall back to a plausible-looking placeholder. Every exclusion is tallied and disclosed, not dropped silently.

How it is checked

83 tests pass and the type-check is clean. The headline sentence is rendered from the committed run data, never hand-typed, and a reproduce command re-derives the committed output from the traffic snapshot, offline, byte for byte. The source, data, and tests are public: read it and run it yourself.