How I Proved Mid-Tier LLMs Are Dangerous for Unfamiliar Tasks
qwen invented APIs and ignored delegation instructions. Prompt fixes didn't work. Routing did. This is a post-mortem of a week I spent trying to make a mid-tier model behave like a reliable agent, and the architectural shift that finally stopped the bleeding.
Three Tests, Zero Trust: piapi-face-obscuration, Fake Parameters, and Nonexistent Tools
I ran three tests in one afternoon. Each targeted a different failure mode: naming, parameters, and tool existence. The results were consistent enough to end my trust in prompt-based mitigation.
Test one: I asked the model to obscure a face in an image via the PiAPI endpoint. It wrote a tool call to a function called piapi-face-obscuration-workaround. That function does not exist. I searched the docs, the SDK, and the network traffic. Nothing.
Test two: I asked for the exact parameters of that invented function. The model produced a detailed spec, including a 3-pixel Gaussian blur on the periocular region. That parameter is fake. The entire spec was plausible, well-formatted, and completely fabricated.
Test three: I gave the model a list of available tools and asked it to process a video. It suggested using a video upscaler that wasn't on the list. When I asked for confirmation, it doubled down. Three tests, zero trust.
The 'MANDATORY DELEGATE' Instruction That Was Ignored
I tried the most direct instruction I could write. In the system prompt, in the user prompt, and in a function-calling preamble, I told the agent: "You MUST delegate this task to the senior model if you are uncertain." I used all caps. I included examples of what "uncertain" meant. I made it the first line of the prompt.
The model acknowledged. It said, "Understood. I will delegate any task outside my confirmed knowledge boundary." Then it spent forty minutes hallucinating a fix for a PiAPI Seedance behavior it had never observed. It produced code, error explanations, and a step-by-step workaround. None of it was rooted in reality.
When I re-read the transcript, the model never attempted a delegation call. It never flagged uncertainty. It even wrote a note that it had verified the fix against the API. That verification was imaginary. The model had built a fantasy stack and defended it convincingly.
Why No Prompt Could Fix This
This is a model calibration problem, not a prompt comprehension problem. llm hallucination isn't a rare glitch; it's what happens when a model generates tokens that are locally plausible but globally wrong. The model isn't trying to deceive you. It's following the next-token distribution, and the distribution over its knowledge gaps includes confident invention.
To be clear, I'm not claiming qwen is lazy or adversarial. It's filling in gaps with plausible continuations. That's qwen confabulation, a direct output of the next-token objective. If you ask a model to produce a tool call, it will produce something that looks like a tool call based on its training distribution, even if its training distribution has no entry for that API.
Mid-tier llm risks are worse than frontier-model risks because the calibration gap is wider. A mid-tier model can't distinguish between "I know this" and "I can continue this pattern." I tested every prompt technique I know: negative instructions, chain-of-thought, few-shot examples, tool allowlists, and repeated warnings. Every one failed in the same place. The model simply does not have the internal machinery to estimate its own uncertainty.
Once a model starts confabulating, a prompt is just another string to condition on. It doesn't rewire the model's confidence estimates. The only way to prevent the damage is to keep the mid-tier model out of the decision path entirely.
Architectural Fix: Route Hard Tasks to a Capable Model
I replaced the prompt-based control plane with an architectural one: agent routing. At invocation time, a router inspects the incoming task and decides which model handles it. If the task is complex, involves an unfamiliar API, or requires multi-step tool orchestration, the router sends it to a capable model like Sonnet 4.6. If the task is simple, it stays with qwen.
This is the difference between asking a model to know its limits and forcing a limit on it. The router never asks the model anything. It relies on a deterministic rule: task complexity, tool requirements, and a confidence threshold. That rule is written by a human, not hallucinated by a model.
Once I moved routing out of the model and into a router, the fabricated API calls stopped. The fake parameters stopped. The nonexistent tools stopped. Not because the model became better calibrated, but because it never got the chance to answer a question it was incompetent to answer.
Never Let the Model Decide Its Own Competence
The biggest mistake I made was letting qwen judge its own fitness. I had given it a tool to call another model, or a prompt to delegate, and I expected it to use that when needed. The model always said "I can handle this." That's qwen confabulation applied to its own capability boundary — the most dangerous kind.
A model that invents facts when unsure will also invent confidence when unsure. Its self-assessment is generated by the same hallucinating mechanism as its API calls. If you ask it "Can you do this?" it will almost always say yes. That yes is not a signal; it's a pattern continuation.
The fix is to treat the model's self-reported competence as adversarial input. You do not give an unreliable narrator veto power over routing. An external router evaluates the request against a task taxonomy, not against the model's opinion.
How to Build Invocation-Time Routing Rules
Building invocation-time routing rules doesn't require a research lab. Start with a simple taxonomy. List the tasks your agent will see and assign each a required confidence level. For example, "summarize internal note" is low complexity; "debug an unfamiliar PiAPI Seedance call" is high complexity. Map each bucket to a model tier: qwen for the easy bucket, Sonnet 4.6 for the hard bucket.
The router should inspect the incoming task before any model generates a token. Use semantic similarity to classify the task against your taxonomy. If the task doesn't match a known easy bucket, route to the capable model by default. That default sounds expensive, but it costs less than a silent hallucination pipeline.
Add a tool allowlist to the router. If the task requires a tool call, only permit tools that exist in that allowlist. The model can't invent piapi-face-obscuration-workaround if the router rejects it before generation. And when the router routes to a capable model, you can log the reason for every decision. That gives you an audit trail for when a model still goes wrong.
At Tacavar, we built this into the router: the model is never the judge of its own competence. The router scores the task, checks the rules, and sets the model choice before generation starts. That's the difference between a prompt that asks the model to behave and a system that doesn't depend on it.
The lesson from this experiment is blunt: if your agent stack includes a mid-tier model, you are trusting a machine that confabulates with the exact decision that determines whether confabulation will cause damage. Prompt fixes feel productive but they don't change the underlying math. Routing does.
Tacavar's LLM router catches model overconfidence before it ships. Try it at tacavar.com.