I Nerfed Mid-Tier LLMs Out of the Routing Decision
I told qwen 'you MUST delegate' in all caps. It said 'understood' — then hallucinated a fix for forty minutes. That was the moment Tacavar's agent stack stopped trusting mid-tier models to know their own limits. We had built a delegation instruction into the system prompt. We had given examples. We had made it mandatory. qwen-plus ignored all of it, invented a fake skill called piapi-face-obscuration-workaround, and confidently offered to run a nonexistent tool. This is agent delegation failure, and it is silent. It looks like work. It produces nothing.
The all-caps delegation instruction qwen ignored
At Tacavar, we run a multi-model agent stack. Mid-tier models handle routine tasks. High-tier models handle hard ones. The routing decision—who handles what—used to be made by the model itself. We asked qwen-plus to assess whether it could debug unfamiliar PiAPI Seedance behavior. We told it, in all caps, 'MANDATORY DELEGATE' if the task was outside its knowledge. It replied 'understood.' Then it spent forty minutes hallucinating. This is not a prompt formatting problem. It is a calibration problem. qwen hallucination at the routing layer means the model does not know what it does not know. It fills gaps with plausible invention. For an operator, that is worse than a crash. A crash is loud. A hallucination is quiet and expensive.
Fake skill names and invented 3-pixel blur parameters
The confabulation was specific. In three tests, qwen-plus invented a skill name: piapi-face-obscuration-workaround. It invented a numeric parameter: a 3-pixel Gaussian blur on the periocular region. It offered to run tools that do not exist. It did this with high confidence, in fluent prose, without hedging. Mid-tier model reliability fails exactly where you need it most: unfamiliar technical tasks. If your agent stack has a routing decision made by a model that fills knowledge gaps with invention, you have a silent hallucination pipeline masquerading as an agent. Tacavar's tests proved this. No prompt engineering fixed it. We tried rephrasing, adding examples, increasing temperature constraints, lowering temperature. Nothing worked. The model's self-assessment was broken.
Why prompt engineering cannot fix model calibration
Calibration is the ability to estimate your own competence. Prompt engineering can shape output format, tone, and even reasoning steps. It cannot install a truthful internal sensor for 'I don't know this.' When qwen-plus encountered unfamiliar PiAPI Seedance behavior, it did not have a knowledge gap detector. It had a language model. So it generated a fix that sounded right. This is llm routing calibration at its core: you cannot prompt a model into accurate self-knowledge. You can only route around it. At Tacavar, we spent weeks trying to fix this with prompts. We wrote delegation trees. We wrote meta-instructions. We wrote 'if you are unsure, say so.' The model said 'understood' and then did the opposite. That is not a bug you can prompt away. It is an architectural mismatch. Capability and calibration are different axes. A model can be highly capable and still poorly calibrated. Prompt engineering can improve capability by guiding reasoning. It cannot improve calibration because calibration is a property of the model's training and architecture, not its instructions. You can teach a model to say 'I don't know' in a specific format, but you cannot teach it to know when it doesn't know. That is why Tacavar's tests with qwen-plus were so damning: the model was capable enough to generate plausible fixes, but not calibrated enough to recognize they were fake.
Invocation-time routing beats model self-assessment
The fix was architectural. Instead of asking the model to decide whether it can handle a task, we moved the decision to invocation time. Tacavar routing now classifies the task before any model sees it. Hard tasks go to Sonnet 4.6. Routine tasks stay with mid-tier models. The model never gets a vote. This is invocation-time routing. It beats self-assessment because it does not rely on the model's internal honesty. It relies on external signals: task novelty, tool requirements, failure history, and complexity heuristics. When we switched to external routing, the hallucinated-fix failure mode stopped appearing: the model simply never gets the chance to route itself. Not because qwen got smarter, but because we stopped asking it to judge itself. For founders building agent stacks, this is the key insight: your router must be separate from your worker. A model that cannot reliably say 'I don't know' cannot be trusted to route. For example, Tacavar's router flags tasks that mention specific APIs not seen in the training corpus, or require tools that are not registered in the agent's skill list. It also measures task depth: single-step vs. multi-step, known vs. unknown domain. These signals are cheap to compute and highly predictive. They do not require the model to reflect on its own knowledge.
Hard tasks go to Sonnet 4.6; never let qwen decide
At Tacavar, the policy is simple. Hard tasks go to Sonnet 4.6. Never let qwen decide whether it can handle something. This is not a knock on qwen. It is a recognition that mid-tier models are optimized for throughput and cost, not for calibration on unfamiliar technical work. Sonnet 4.6 handles the hard stuff because it has better knowledge boundaries and more reliable reasoning under uncertainty. But even Sonnet 4.6 should not be the router. The router is Tacavar's invocation-time layer. It looks at the task, the tools, the context, and the history. It routes. Then the selected model works. This separation of concerns is what makes the stack reliable. If you let a mid-tier model decide, you get qwen hallucination on your critical path. If you let a high-tier model decide, you pay high-tier costs for every routing decision. Invocation-time routing is the only approach that scales. The cost difference is real, but so is the cost of a forty-minute hallucination. Invocation-time routing adds one classification step before inference. The savings from avoiding hallucinated fixes are orders of magnitude larger.
A routing checklist for agent stacks
Here is the checklist Tacavar uses for every agent stack we deploy. Use it as a template.
- Classify task difficulty before model invocation. Do not ask the model to classify itself.
- Route hard tasks—unfamiliar APIs, novel debugging, multi-step tool use—to Sonnet 4.6 or equivalent.
- Never let a mid-tier model decide whether it can handle something. Its self-assessment is not calibrated.
- Log every routing decision with the signals that drove it. Audit for silent hallucination pipelines.
- Test your router with adversarial tasks: fake APIs, unusual parameters, missing documentation. See if your mid-tier model confabulates.
- Separate routing from execution. The model that does the work should not be the model that chose the work.
- Set a cost ceiling for high-tier routing. Use invocation-time heuristics to keep routine tasks on mid-tier.
- Monitor for agent delegation failure: tasks that take too long, produce plausible but unverifiable output, or reference nonexistent tools.
- Re-calibrate your router as your stack evolves. New tools, new APIs, new failure modes.
- Never trust a model that says 'understood' without evidence. Trust your router instead.
This checklist is not theoretical. It comes from Tacavar's own failures. We nerfed mid-tier LLMs out of the routing decision because they could not be trusted to know their limits. The result is a stack that routes hard tasks correctly, every time. Start with the router. The model is replaceable. The routing logic is the product.
The core lesson: mid-tier model reliability is a function of task familiarity. For unfamiliar technical tasks, it collapses. Your routing layer must be external, invocation-time, and model-agnostic. Tacavar routing does this. It is the difference between an agent stack that works and one that hallucinates for forty minutes while you wait.
Route hard agent tasks with Tacavar's invocation-time model router at tacavar.com/routing.