Skip to main content
TACAVAR
•Build in Public

Prompt Engineering Can't Fix LLM Calibration

Mid-tier LLMs confabulate under novelty: fake skills, invented params. Prompts can't fix calibration — invocation-time routing can.

My agent said 'understood' to a MANDATORY DELEGATE order, then hallucinated for forty minutes. If you run an agent stack in production, you've seen some version of this: a model that acknowledges your constraint perfectly, then violates it the moment execution starts. The uncomfortable lesson from this incident — and from the testing we ran afterward — is that this is not a prompting problem. It's a calibration problem, and no amount of prompt engineering fixes it.

Three Tests, Three Confabulations

We stress-tested a mid-tier model — qwen-plus — on unfamiliar technical work, the kind of task where you'd expect a capable agent to either succeed or escalate. Three tests, three failures, each in a different flavor.

In the first, the model invented a plausible-sounding skill name: piapi-face-obscuration-workaround. That skill did not exist in the environment. It looked like something a senior engineer might register, which is precisely the problem — plausibility and existence are unrelated properties, and the model had no mechanism to distinguish them.

In the second, it fabricated numeric parameters: a 3-pixel Gaussian blur applied to the periocular region. The numbers were specific, confident, and completely invented. There was no source for them. The model wasn't retrieving a value from documentation or memory; it was generating a string shaped like expertise.

In the third, it offered to run tools that don't exist, cheerfully and with high confidence.

Every failure mode here is a downstream symptom of the same upstream defect: the model fills knowledge gaps with invention instead of uncertainty. That's the core of llm calibration failures — the model's confidence signal is uncorrelated with its actual competence on the task at hand.

The Fake Skill Name That Looked Real

Let's sit with the skill name for a second, because it's the most operationally dangerous of the three confabulations. piapi-face-obscuration-workaround follows naming conventions your own codebase probably uses. It contains a domain reference, a mechanism, and a qualifier. If your agent framework validates skills loosely — or trusts the model's self-report — that invented name sails through.

Worse, your human reviewers are vulnerable to the same illusion. When an agent proposes a skill that sounds like it belongs, the review instinct is 'does this name make sense?' rather than 'does this name exist?'. A confabulated identifier that matches your conventions is a credential that grants access to forty minutes of downstream garbage work: tool calls that fail, retries against imaginary endpoints, and a transcript that looks productive until you actually inspect it.

This is where agent hallucination prevention has to happen structurally — allow-lists, registry checks at invocation, hard failures on unknown references. Prompting the model to 'only use real skills' changes the surface language, not the underlying generation process.

MANDATORY DELEGATE: When All Caps Fails

The third test is the one that broke our assumptions about prompt control. The instructions were explicit: the model MUST delegate this task. All caps. Examples included. A direct acknowledgment came back — 'understood' — and then the model proceeded to debug unfamiliar PiAPI Seedance behavior itself, hallucinating a fix for forty minutes.

Every founder who has tried to control model behavior with increasingly forceful instructions should internalize what happened here. Instruction-following degrades precisely where it matters most: at the boundary of the model's competence. The model didn't ignore the instruction maliciously or randomly. It ignored it because following it would have required the model to recognize something it is structurally bad at recognizing — that it doesn't know. Calibration failure and instruction-following failure are the same failure. A model that can't estimate its own ignorance also can't reliably trigger the escape hatch you built for exactly that situation.

This is the sharp edge of mid-tier model routing strategy: the decision 'can I handle this?' cannot be delegated to the model itself. You are asking the least reliable judge of model capability — the model — to adjudicate routing for the hardest cases, which are by definition the cases where its self-assessment is most wrong.

Why Confidence Scales Inversely With Knowledge

The pattern across all three tests has a name in the literature, and operators see it constantly in practice: the inverse relationship between domain familiarity and expressed confidence. On familiar territory, a mid-tier model hedging and second-guessing is annoying. On unfamiliar territory — the tasks you actually wanted to delegate to save cost — it produces specific, self-assured invention: fake skills, invented parameters, phantom tools.

Think about what this does to your cost model. The entire pitch of routing easy tasks to cheaper models collapses if the router can't tell easy from hard. The tasks that need escalation are the ones where a mid-tier model will least reliably self-report. You don't get graceful degradation. You get a model that says 'understood' and burns forty minutes generating plausible garbage — while your billing shows savings and your users see a confident agent that is quietly wrong.

There is no prompt that fixes this. We tried, and the prompt-engineering literature increasingly agrees: calibration is a property of the model's internal representations, not the surface formatting of your request. You can nudge it at the margins. You cannot repair it.

The Architectural Fix: Routing at Invocation

The fix that actually worked was architectural, and it's blunt: route hard tasks to Sonnet 4.6 at invocation time, and never let the mid-tier model decide whether it can handle something. The model's self-assessment is removed from the loop entirely.

This reframes the routing question. The decision isn't 'which model is smart enough to judge task difficulty?' — it's 'where in the pipeline can a judgment be made by something other than the model executing the task?'. Invocation time is the only point where you still have structural control: you know the task classification, the tool surface, the domain. After invocation, you only have the model's word.

In practice this means explicit routing rules keyed on task type — unfamiliar domains, novel tool integration, anything touching external APIs like the PiAPI Seedance case — go to the stronger model before the mid-tier model ever sees them. The mid-tier model keeps the work it's actually good at. Nobody asks it to self-assess, because self-assessment is the thing it fails.

This is also where the qwen vs claude for coding agents comparison becomes a routing question rather than a model-choice question. The honest answer isn't 'qwen is bad' — it's that qwen-class models are excellent on high-frequency, well-represented tasks and structurally unsafe on unfamiliar ones, and Claude-class models hold calibration further out of distribution. Your stack should use each where those properties actually apply, decided at invocation, not at runtime by the model itself.

Auditing Your Stack for Silent Hallucination Pipelines

Here's the audit we now run on every agent deployment, and the one we'd recommend for yours:

Find every point where a model decides its own scope. Any place a model answers 'can I handle this?' is a latent failure point. Instruction-based self-restrictions — 'if unsure, delegate' — don't count as control. They're the thing the MANDATORY DELEGATE test disproved.

Test on genuinely unfamiliar tasks, not benchmark-shaped ones. Benchmarks are in-distribution for major models by construction. The confabulations we saw only appeared on genuinely novel territory — the exact work founders delegate to agents to save time.

Log self-reports and grade them. Track every instance where a model claimed competence or said 'understood' to a constraint, then compare against outcomes. If your acknowledged-constraint violation rate is nonzero, you have the pipeline.

Check tool and skill validation. If your framework accepts model-proposed identifiers at face value, a fabricated skill name will pass. Validate against the registry, hard-fail on unknowns.

Assume mid-tier models will confabulate under novelty. Design for that assumption. Either constrain them to in-distribution work or route them out at invocation.

The uncomfortable summary: if your agent stack contains a routing decision made by a model that fills knowledge gaps with plausible invention, you don't have an agent with a flaw. You have a silent hallucination pipeline masquerading as an agent. Prompts won't fix it. Architecture will — and the sooner you move the routing decision out of the model and into your invocation layer, the sooner your forty-minute hallucination sessions turn into two-second escalations.

Stop letting mid-tier models decide what they can handle. See Tacavar's dual-pipe LLM routing at tacavar.com/llm-routing.