Cost-Per-Success: The LLM Routing Metric Nobody Measures
Cheap-first routing can show 79% invoice savings and still cost 3x more per success. Cost-per-success catches it: the arithmetic and the instrumentation.
Every LLM bill is itemized by call. Every routing decision is judged against that bill. The number almost nobody computes is the only one that decides whether the routing worked: total spend to reach a correct outcome, counting every retry, escalation, and fallback that happened along the way.
The industry has spent two years optimizing cost-per-call — token pricing, model selection, cascade order. A recent operator discussion on Hacker News named the gap directly: nobody measures cost-per-success. That the observation still reads as novel in 2026 is the telling part. The metric is absent from standard LLM observability to the degree that its existence is a forum post, not a dashboard.
What cost-per-success means
Cost-per-call is the price of one inference. It is on every provider invoice, which is why it became the industry's default optimization target: it is the number that is easy to see.
Cost-per-success is the number that matters:
Cost-per-success = total spend across all attempts on a task ÷ correct, usable outcomes
The numerator is cumulative: the cheap call that failed, the retry that failed differently, the escalation to the capable model, the fallback after that, and the human rework when the chain still does not land. The denominator counts only outcomes that were actually right.
The reason nobody measures it is structural, not laziness. Observability layers are built per request; economics happen per task. Your tracing dashboard records that request 4 succeeded. It does not record that request 4 only existed because requests 1 through 3 failed, and that the sum of all four is the real price of that success. Until the task — not the call — is the unit of account, cost-per-call stays the only visible number, and teams optimize what they can see.
The arithmetic: how a retry chain flips the verdict
Here is a worked example with illustrative but production-shaped numbers. The task: structured extraction over messy documents. The batch: 1,000 tasks.
Setup. A capable model handles this task type at $0.020 per call with 98% correctness. A cheap model charges $0.002 per call — ten times cheaper — and succeeds 60% of the time on this task type. Published routing analyses put 60–70% of general queries safely within reach of smaller models, but the routable share collapses to 20–35% for reasoning-heavy and code-adjacent work; extraction with edge cases sits at exactly the contested margin where routing decisions are actually made. Every failure that is eventually caught costs $0.60 in downstream human rework. Assume one retry on the cheap model before escalation.
Route A — go straight to the capable model.
- 1,000 calls × $0.020 = $20.00 in inference
- 20 failures (2%) × $0.60 = $12.00 in rework
- Total: $32.00 for 1,000 correct outcomes → $0.032 per success
Route B — cheap-first, perfect failure detection.
- Attempt 1: 1,000 × $0.002 = $2.00 → 600 correct, 400 failures detected
- Retry: 400 × $0.002 = $0.80 → 240 correct, 160 detected
- Escalate: 160 × $0.020 = $3.20 → 157 correct, 3 reworked ($1.80)
- Total: $7.80 for 1,000 correct outcomes → $0.0078 per success
The cascade wins by 4×. This is the outcome the routing research promises: FrugalGPT (Chen et al., Stanford, 2023, arXiv:2305.05176) matched the performance of the best individual model with up to 98% cost reduction using learned query cascades. When failures are detected, cheap-first is close to free money.
Route C — same cascade, validator catches 70% of failures.
- Attempt 1: 1,000 × $0.002 = $2.00 → 600 correct; 400 failures, 280 flagged, 120 ship silently
- Retry: 280 × $0.002 = $0.56 → 168 correct; 112 failures, 78 flagged, 34 ship silently
- Escalate: 78 × $0.020 = $1.56 → 76 correct, 2 reworked
- Human rework: 156 total failures × $0.60 = $93.60
- Total: $97.72 for 1,000 correct outcomes → $0.098 per success
| Strategy | Inference spend | Rework | Total | Cost per success | What the invoice says |
|---|---|---|---|---|---|
| Capable model direct | $20.00 | $12.00 | $32.00 | $0.032 | baseline |
| Cheap-first, 100% detection | $6.00 | $1.80 | $7.80 | $0.0078 | "70% savings" |
| Cheap-first, 70% detection | $4.12 | $93.60 | $97.72 | $0.098 | "79% savings" |
Read the last row twice. By the invoice — the only dashboard most teams have — Route C is the best outcome on the page: lowest inference spend, highest reported savings. By the task ledger, it costs 3× more than paying for the capable model upfront. The cheap model did not just fail to save money. It tripled the bill.
Solve for the break-even and the real variable appears: on these assumptions, the validator has to catch roughly 92–93% of failures before cheap-first beats direct routing at a 10× price advantage. The verdict flips on detection rate, not on price. A tenfold cost advantage is erased by missing roughly one failure in thirteen.
What the number is actually measuring
This is the uncomfortable conclusion: cost-per-success is primarily a measurement of your failure detection, not your model selection. Routing presents as a model-choice problem. It is a detection problem. The team arguing over which model to route to is usually avoiding the question of whether they would even know if the cheap one were wrong.
We have written the cost-floor side of this ourselves. A routing audit of our own stack found the entire multi-agent fleet running on the free-tier efficient preset — a $0 inference bill, later documented down to the droplet. Those numbers were real, and cheap routing can genuinely win, as Route B shows. But the audit checked where calls went and what they cost. It did not check what fraction of outcomes was correct, because nothing in the stack recorded outcomes at all. That is the standard state of the industry: the invoice is audited to the cent, the denominator is unaudited entirely. Cost-per-success is the metric that closes that gap — see Why Agent Routing Matters More Than Prompt Engineering for the reliability argument, and our cost floor and zero-cost inference posts for how low the numerator can go when routing is actually deliberate. The open question both leave is the denominator.
The instrumentation sketch
Nothing about this requires a re-platform. If your stack propagates a trace ID, it can propagate a task ID. The sketch:
calls (one row per model call)
task_id, attempt, model, tokens_in, tokens_out,
cost_usd, latency_ms,
exit_status in {ok, schema_error, validator_flag, escalated}
outcomes (one row per task, written at termination)
task_id, task_type, route_taken, attempts,
cumulative_cost_usd, wall_clock_ms,
final_status in {success, failed, human_reworked, abandoned}
Three implementation notes. First, cumulative_cost_usd is written once, at task termination, as the sum of the task's call rows — never incremented per call, or you will double-count retries under concurrent escalation. Second, final_status requires an outcome signal: a deterministic check, a critic model, or a human gate. Without it the denominator is fiction and Route C reports itself as Route B. Third, dashboard two numbers per task type and route: median cumulative spend, and cost-per-success against the direct-capable baseline.
Then make the routing respond. When a route's cost-per-success for a task type crosses the direct-capable baseline, stop cascading and go direct — the same logic as a circuit breaker, applied to money instead of failures. When the cheap path stops paying, it trips. This is the economic layer that makes orchestration rational, and it pairs with trust scoring: a cheap model's output is only cheap if its failures are caught, and catching failures is a trust function, not a pricing function.
FAQ
What is cost-per-success in LLM routing? Total spend across every call, retry, and escalation required to produce one correct, usable outcome — as opposed to cost-per-call, the price of a single inference. It is the metric that determines whether a routing strategy saves money or merely moves it into rework.
How is cost-per-success different from cost-per-call? Cost-per-call is per request and always favors the cheapest model. Cost-per-success is per task and prices in the failure chain: failed cheap attempts, escalations, fallbacks, and human rework. A model 10× cheaper per call can cost 3× more per success when its failures are missed.
When does model cascading stop saving money? When the failure detection rate drops below break-even for the task type. In the worked example above, a 10× cheaper model needed roughly 92–93% of its failures caught to beat direct routing once rework was priced at $0.60 per incident. Cascades win decisively with reliable detection and lose decisively without it.
How do you instrument cost-per-success? Propagate a task ID through every call, log per-call cost and exit status, write one terminal outcome row per task with cumulative spend, and divide total spend by correct outcomes per task type and route. It is a schema decision on top of tracing you already run, not new infrastructure.
Is there an industry benchmark for cost-per-success? No — that absence is the point of the metric. The only meaningful baseline is internal: the cost-per-success of routing directly to your most capable model. Any cascade that cannot beat that number is costing you money while your invoice reports savings.
The invoice tells you what you spent. Only the task ledger tells you what you bought. Teams that internalize the difference stop arguing about model prices and start investing in failure detection — because that is where the verdict actually flips. The framework layer that decides routing is worthless without the measurement layer that judges it; see our SDK-vs-framework routing comparison for how that choice gets made.
You built it. We optimize it.