Skip to main content
TACAVAR
•Build in Public

Audit What Your LLM Stack Actually Runs

Tacavar's llm routing audit exposed dead config, a silent Sonnet-to-Ollama fallback, and accidental savings. Audit what your LLM stack actually runs.

My config said Claude. My container was running Qwen. Garbled Vietnamese gave it away. When I sat down to run a proper llm routing audit on Tacavar's agent stack, I expected to find a forgotten $200/month Anthropic bill. Instead I found dead config, a silent downgrade chain, and one accidental win worth real money. Two audits, weeks apart, told the same story: what your .env file says and what your process is actually running are different things.

Two Audits, Two Surprises: Dead Config and Silent Fallbacks

The first audit targeted Tacavar's Bailian deployment — a 12-agent team running around the clock. The config declared qwen. The container was actually running the qwen_efficient preset. Same model family, different preset, and critically, different behavior at the routing layer. A config file and a live process had quietly diverged, and nothing in the stack flagged it.

The second audit hit a Tacavar Hermes profile called "heavy" — built specifically to reach Claude Sonnet 4.6 through CLIProxyAPI. It had never once touched Sonnet. The profile was silently falling back through qwen-plus to a local Ollama model, gemma4:26b, on every call.

Two audits, two failure modes, one shared root cause: nobody had verified the routing layer against reality. That's the case for a recurring llm cost audit — drift cuts both ways. You can silently burn hundreds of dollars a month or silently save them, and in both cases your config file will insist nothing changed.

How router.py Ignores Your Environment Variables

Here's the detail that made the Bailian finding worse: the stack had an escape hatch, and it was dead.

Tacavar's Bailian env included OPENCLAW_BASE_URL=10.116.0.2:18801 on bailian-app-1 — an explicit directive pointing at a specific inference endpoint. It did nothing. The router on Tacavar's stack doesn't read environment variables for routing decisions. It routes by model prefix: anything matching claude* goes direct to the Anthropic API and gets billed; qwen*, glm*, and minimax* route to DashScope under Tacavar's Coder Pro coverage.

So that variable wasn't just stale — it was never wired in. It was documentation wearing the costume of configuration.

The operator-level lesson: environment variables are intent, not enforcement. To verify model routing, don't read the .env file. Read the router source, then the running container — docker exec bailian-app-1 env and the startup logs. The process env is the truth; the file is a rumor about it.

The Fallback Chain That Quietly Downgraded Sonnet to a Local Model

The Hermes failure is the more instructive one, because the config looked right and every casual test passed.

The profile declared model.default: claude-sonnet-4-6 with provider: custom. The custom provider path through CLIProxyAPI silently failed — no error surfaced to the caller. The stack then walked its fallback chain: qwen-plus first, then down to gemma4:26b on Ollama. The request succeeded. The caller had no idea it was now talking to a 26-billion-parameter local model instead of Sonnet.

This is why model fallback chain debugging deserves first-class tooling in any agent stack. Fallbacks exist for resilience, and they're worth having — but a fallback chain without observability is a downgrade machine. Every rung it descends changes your cost profile, your latency, and your capability ceiling, and none of it appears anywhere unless you log which model actually answered.

The fix on Tacavar's side was two lines: call hermes chat -m claude-sonnet-4-6 against main. The -m flag resolves custom providers correctly where the profile-based default did not. Two lines, and a profile that had never touched Sonnet hit it every time.

Why Response Text Can't Validate a Model: Run Tool-Calling Tests

Here's the uncomfortable part: our quality checks passed while gemma4:26b was impersonating Sonnet.

When asked "what model are you," gemma4 played the role convincingly — it mimicked Sonnet's tone and formatting well enough that a reviewer reading transcripts saw nothing wrong. LLMs will role-play being other models, which makes self-report worthless as a routing check. So is stylistic similarity.

What exposed it was a tool-calling test. Under structured output constraints — real function schemas, verifiable argument correctness — the local model broke. Tacavar's logs caught garbled Vietnamese tokens on tool calls: output like ícia lính lính lính where structured JSON belonged. Hallucinated tool results and tokenizer breakage under schema pressure are the smoking guns. A small local model can imitate prose; it cannot reliably imitate Sonnet emitting valid tool arguments.

The rule in Tacavar's playbook: never validate a route with response text. Build a canary that runs a deterministic, schema-validated tool call against every routed profile on a schedule. If the output doesn't parse or the arguments are wrong, you're not talking to the model you think you are.

The Lucky Find: Auditing Your Way Into Silent Savings

Now the good news from Bailian — and why it scared me more than the bad news.

Because the container was running qwen_efficient and the router matches on the qwen* prefix, all 12 agents still routed to DashScope under Tacavar's Coder Pro coverage. Zero Claude routing. Zero Anthropic Extra Usage burn. Across a 12-agent team running continuously, that's a meaningful monthly number that stayed at zero purely because the drifted preset name happened to still match the free side of the prefix split.

That's luck, not architecture. If a future refactor renamed the preset to something without the qwen prefix — bailian-fast, say — the identical drift would have flipped all 12 agents to billed Anthropic routing overnight, and the config file still would have said qwen. An llm cost audit finds savings in both directions, and savings found by accident are savings you can lose by accident. The audit's real output wasn't "we're saving money" — it was "our savings are unguarded." We codified the routing assumptions immediately after.

A Routing Audit Checklist for Any Agent Stack

Run this on Tacavar-shaped stacks or anything resembling one:

  1. Diff declared config against the live process. Compare your .env and profile files against docker exec <container> env and startup logs. The file is a claim; the process is the fact.
  2. Read the router source, not the docs. Find the actual decision logic — prefix tables, provider maps — and confirm every declared model string resolves through it. Grep for env vars the router ignores.
  3. Force a fallback on purpose. Take your primary provider offline in staging and log which rung of the chain answers. If nobody knows, that's the finding.
  4. Run schema-validated tool-calling canaries per profile. Deterministic task, verifiable output, on a schedule. Response-text quality checks don't count.
  5. Grep logs for fallback events and unexpected model names. Any model string you didn't declare is a silent downgrade or a silent bill.
  6. Reconcile invoices against route distribution. Your usage dashboard and your per-model route counts should agree line by line. Discrepancies are config drift with a price attached.
  7. Alert on identity mismatch. Where the API exposes the serving model, log it and compare it against the requested model on every call.
  8. Re-audit after every change. Dependency bumps, preset renames, env edits — agent stack config drift re-enters silently through exactly these doors, and it never announces itself.

Config that matches reality is a coincidence you maintain, not a state you achieve. Get a routing and cost audit methodology modeled on Tacavar's stack at tacavar.com.