Agent Token Cost Optimization Playbook: Context Budgets, Caching, and Model Routing

The first invoice after you put an agent pipeline into production is always a surprise. Not because the per-token prices were misquoted, but because the architecture multiplies them: a planner that re-reads the full conversation every turn, a router that sends trivial classification calls to your most expensive model, a retrieval step that pastes forty pages into context to answer a one-line question. Token spend in an agent system is an architectural property, not a pricing problem. This playbook covers the four levers that actually move it, in the order you should apply them.

We measured the effect of each lever against a real multi-agent pipeline (a research-and-write workflow with a supervisor, two specialist workers, and a critic pass). Baseline cost per completed run: roughly $0.40. After all four levers: $0.11, with no measurable drop in task success rate. Your numbers will differ; the sequence will not.

Lever 0: Instrument Before You Optimize

Every optimization below is a guess until you can attribute spend to a stage. The minimum viable telemetry is per-stage token counts (input and output, separately), per-stage model name, and per-run cost. LangSmith, Helicone, or a forty-line wrapper around your provider calls all work. What matters is that you can answer "which stage spent 80% of the money on run #412?" in under a minute. In our baseline, one stage, the critic pass, was consuming 46% of total spend while rarely changing the final answer. We would never have known without per-stage attribution, because the critic was the cheapest model in the stack — it just ran three times per run with the entire accumulated context each time.

If you have not already, define the metric you are protecting before touching anything: cost per completed task, not cost per call. A cheaper call that fails twice as often is a cost increase. We have written up why per-task beats per-token as the unit of account in our agent cost-per-task benchmark, and it is the frame the rest of this playbook assumes.

Lever 1: Context-Window Budgeting

The single largest line item in most agent pipelines is re-read context: every stage receives the full conversation history, the full retrieved corpus, and often the full outputs of every sibling stage. Input tokens dominate because they are re-billed every call. The fixes are unglamorous:

• Summarize the conversation, not the message. Beyond roughly ten turns, replace the raw transcript with a rolling summary plus the last few verbatim messages. A two-paragraph summary written by a mid-tier model costs a fraction of re-feeding forty messages to a frontier model — and long-context degradation means the frontier model was extracting less from those messages anyway.
• Budget retrieval, do not dump it. Cap retrieved context at what the task plausibly needs. For a one-line factual lookup, three chunks beats thirty. Every chunk you add past the answer costs input tokens on every subsequent turn of the run.
• Pass outputs, not transcripts, between agents. When a worker agent reports to a supervisor, it needs the conclusion and the key evidence, not its own 6,000-token scratchpad. Enforce a structured, bounded report format (a schema with a hard character cap) at the handoff boundary.
• Set a hard context ceiling per stage. Decide the maximum context each stage is allowed to consume, and make exceeding it a logged event rather than a silent invoice. Ceilings turn runaway stages from surprises into alerts.

Context budgeting alone removed 38% of our baseline spend, more than any other lever. It also cut p95 latency, because providers price and process long contexts slower. This is also where checkpointing discipline pays a second dividend: a checkpointer that stores state compactly lets a resumed run rebuild context from a checkpoint instead of replaying the whole transcript. Our LangGraph checkpointing guide covers the storage side of that pattern.

Lever 2: Prompt and Response Caching

Caching in agent pipelines comes in three layers, and most teams stop at the first:

• Provider prefix caching. Anthropic, OpenAI, and DeepSeek all discount repeated prompt prefixes. It requires zero code changes and is nearly free money — but only if your prompts are deterministic enough to share long prefixes. Move the stable content (system prompt, tool definitions, few-shot examples) to the front of the context and the variable content (user turn, retrieval results) to the end, or you fragment the cache with every call.
• Application-level result caching. Embeddings for classification, deterministic tool outputs, and repeated factual lookups should be memoized at your layer. An embedding cache on a 10,000-document pipeline turns a $30 classification pass into a few dollars after the first run.
• Semantic caching. For user-facing agents, cache answers by embedding similarity, not string match. A support agent that answers "how do I reset my API key?" the same way for 4,000 phrasings should pay for one generation, not 4,000. Set a similarity threshold conservatively (high enough that a cached answer is almost always correct) and log cache hits as their own metric so you can watch quality drift.

Caching contributed 27% of our savings, mostly from prefix-cache hits once we stabilized prompt structure. One warning: caches hide bugs. A stale cached tool output or an outdated cached answer is indistinguishable from correct until a user complains. Version your cache keys by prompt template and by tool schema, and treat cache hit rate as a metric with an expected band, not a number to maximize.

Lever 3: Model-Tier Routing

Not every stage deserves the same model, and the frontier model in your stack is almost certainly doing work a mid-tier model handles identically. The routing decision should be per-stage and evidence-based: run a sample of each stage's real traffic through a cheaper model, score both outputs against your task rubric, and demote the stage only if quality holds. In our pipeline, three of six stages survived demotion to a model at roughly one-fifth the token price with no rubric-score change; the planner and the critic did not, and stayed on the frontier tier.

Three routing rules that hold across pipelines:

• Classify cheap, reason expensive. Intent classification, routing, extraction, and format compliance are pattern-matching tasks. Reasoning about ambiguous goals, multi-step planning, and final quality judgment are not. Most pipelines invert this.
• Route on difficulty, not on stage name. A "hard" stage with easy inputs should still take the cheap path. Score input complexity (length, ambiguity flags, prior-turn failure signals) and route per call, not per stage.
• Keep an escape hatch. Any cheap-model output should be cheap to verify. Where a verifier exists, use it; where it does not, reserve the expensive model for a fraction of traffic as a quality canary so you detect tier degradation before users do.

Model-tier routing contributed 24% of our savings. The failure mode to design against is silent quality decay: a demoted stage whose rubric score drifts down one point a month will never trigger an alert but will erode the product. Re-score demoted stages monthly against a fixed evaluation set.

Lever 4: Kill the Wasted Runs

The cheapest tokens are the ones you never spend. Most production pipelines carry a baseline of doomed runs: retries that were never going to succeed, agents that loop on unanswerable inputs, and speculative parallelism that burns five branches to keep one. Two controls cover most of it:

• Cap retries with backoff on failure type. Retry transient provider errors aggressively (they are cheap and usually succeed) and permanent errors never (a malformed input will fail identically every time). Cap total attempts per run and make the cap visible in telemetry.
• Detect loops early. Track repetition: identical tool calls, identical assistant messages, or context that stops changing across turns. A loop detector that terminates at turn three instead of turn twenty saves the full tail of a run, and that tail is where costs explode.

This lever contributed the smallest share of savings (11%) but the largest share of latency improvement, because doomed runs are also slow runs. It pairs naturally with circuit breakers, which we covered separately in our circuit breaker patterns article.

The Order Matters

Apply the levers in this order and each one makes the next one cheaper to evaluate: instrumentation tells you where context bloat lives; context budgeting shrinks every subsequent call; caching exploits the now-stable prefixes; routing demotes stages whose cheap-model quality you can verify; and loop termination mops up the tail. Teams that start with routing, the most seductive lever, routinely pay frontier-model prices for bloated context they could have removed for free.

The meta-point: token cost optimization is not a one-time project. Models repriced, prompt templates drift, and usage patterns shift as users discover new features. The instrumentation from Lever 0 is the permanent asset; everything else is a quarterly re-run of the same playbook against current traffic.