What 17,000 Agent Runs Reveal About Tool Choice
Armature measured 16,893 coding-agent sessions across Claude Code, Codex, and Cursor. What the data says about how agents actually pick tools.
Everyone benchmarks agents on whether they finish the task. Almost nobody measures the decision underneath the task: which tool the agent picks when it has to pick one, and why.
That decision is quietly becoming the most important routing layer in your stack. Vercel reported in April that over 30% of its deployments were initiated by coding agents — up 1,000% in six months. When a third of your infrastructure choices are made by software, the question stops being "which agent is smartest" and becomes "which agents do my agents trust, and on what basis."
A team at Armature ran the largest measurement of agent tool choice we have seen. They watched 16,893 sessions — three coding agents (Claude Code, Codex, Cursor), 75 synthetic repositories across 10 languages, 1,163 prompt variations, and four operator personas from vibe-coder to enterprise engineer. Agents didn't just recommend tools. They implemented them, and every trace is public.
The results are less flattering to agents than the benchmarks suggest — and more useful.
Agents disagree more than they agree
Across identical categories, all three agents picked the same tool in only 42% of cases. In voice agents, Claude Code picked Twilio, Codex picked the OpenAI Realtime API, and Cursor picked Vapi. Three agents, three answers, same question.
If you assumed running the same task through two agents gives you a redundant check, the data says otherwise. You get two independent opinions that often diverge — which is either a liability or a free ensemble, depending on whether your pipeline can hold the disagreement instead of collapsing it.
Each agent has a different epistemology
The disagreement isn't random. It's structural — each agent gathers evidence differently:
- Codex almost always searches the web (94% of sessions), but in 9 out of 10 queries it constrains itself with operators like
site:auth0.comto force the answer through a domain it already trusts. Codex doesn't browse. It verifies. - Claude Code relies on its priors and searches in only ~30% of sessions — but when it does search, it reads three times more pages than Codex. Where its training data is thin (newer sectors like sandboxes), web use jumps to ~80%. Confidence is a function of prior coverage.
- Cursor sits in between, grounding decisions in the web about two-thirds of the time.
The implication for agent tool choice in production: you are not deploying one decision-maker, you are deploying three epistemologies. Codex inherits whatever its trusted domains say. Claude Code inherits its training cutoff. Cursor inherits the live web, with all its SEO pressure. Armature sells growth services to dev-tool vendors — their commercial interest is precisely in shaping those surfaces — so read the framing accordingly. The underlying traces, however, are public, and the behavioral split is hard to fake.
Context beats brand
Same ask, four repositories, four languages, four different email-provider winners: Resend on TypeScript, Sendgrid on Python, Postmark on Go, Azure ACS on Java. Vercel swept TypeScript repositories and was never recommended on Python, where Render dominated.
Agents don't pick the best tool. They pick the best-shaped answer to the context they can see. Repository language, existing dependencies, and lockfiles did more to determine the outcome than any leaderboard position. If your codebase context points the wrong way, no amount of brand strength will save the pick — and if it points the right way, the agent will argue for you.
Mentioned is not chosen
This is the finding vendors should lose sleep over. LangChain: 194 mentions, 4 picks. Paypal: 139 mentions, 0 picks (Stripe won 124 of those 139 sessions). Adyen: 175 mentions, 3 picks. Netlify: 152 mentions, 6 picks. Supabase: 242 mentions as the most-cited database, still dominated by Neon.
Familiarity gets you into the conversation. It does not get you installed. Agents pattern-match mentions quickly and then discard them against fit — and they are often wrong when they do, because they read vendor pages the way a tired engineer does. Mailgun lost to Postmark partly because agents read "1-day retention" on the free plan and moved on. Supabase lost database picks partly because bundled BaaS pricing made a simple need look complicated. In 388 of the 5,292 valid sessions, platform-management overhead drove the outcome; in 195, cost did — frequently because of presentation, not a disqualifying datapoint.
Documentation is now a conversion surface for agents, not just for humans. Precision in how a tool describes itself changes whether it gets installed.
Where the framework layer falls short
This is where the agent tool choice data meets the production-framework discussion happening in r/AI_Agents and elsewhere: the frameworks most agents run on — the orchestration and agent-runtime layer — have almost no machinery for any of this. They expose tools. They don't model how the agent will weigh them.
What 17,000 runs say an agent's tool surface actually needs:
- Decision telemetry. Not just "tool called" but "alternatives considered, evidence gathered, source consulted." Without it you can't audit why a pick happened — and 42% consensus means the why matters more than the what.
- Context integrity as a control plane. A small repository edit flips the pick. That's powerful for steering, and dangerous if you don't control it. Your README, your lockfiles, and your docs are inputs to your agents' decisions whether you treat them that way or not.
- A deliberate web posture. If your agent searches, it is exposed to whoever optimized the pages it reads. Deciding whether an agent verifies (Codex), reasons from priors (Claude Code), or splits the difference (Cursor) is an architectural choice with vendor-lock-in consequences — and defaults nobody set explicitly.
Our own breakdown of the agent tool tax and dynamic gating covers the cost side of this — when a tool call is worth its token spend and when it isn't. What the Armature data adds is the selection side: the pick itself is already a routed, biased, context-dependent decision before cost ever enters the picture.
And if you're choosing which agent framework to standardize on, the AI agent frameworks compared landscape is the hub — because tool-choice behavior differs by harness, and harness choice is a decision about which epistemology your operation inherits.
What to do with this
None of this is an argument against agents picking tools. It is an argument for treating agent tool choice as a designed surface, not an accident:
- Instrument the picks. Log alternatives considered and sources consulted, not just calls made.
- Own your context. Your docs and repo metadata are agent inputs; write them for the reader that actually makes the decision.
- Use disagreement deliberately. Two agents disagreeing is information. Collapse it only after you've looked at why.
- Set the web posture per task. Verification-heavy for compliance-sensitive picks, prior-heavy for well-trodden ones — and make it a config decision, not a model accident.
Agents already choose your tools. The only question is whether you designed the decision.
You built it. We optimize it.
FAQ
How many runs were in the Armature study? 16,893 sessions across Claude Code, Codex, and Cursor, run in ephemeral sandboxes against 75 synthetic repositories with 1,163 prompt variations. After validity filtering, 5,292 sessions on 51 codebases across 18 sectors formed the published dataset; all traces are public.
How often do Claude, Codex, and Cursor agree on a tool? All three agents picked the same tool in 42% of category cells. In voice agents, for example, they picked three different winners: Twilio, OpenAI Realtime API, and Vapi.
Why does Supabase get mentioned so often but rarely picked? In the study, Supabase was the most-mentioned database (242 mentions) but lost most picks to Neon — partly because bundled BaaS pricing and feature presentation made a database-only need look heavier than it was. Mentions measure familiarity; picks measure perceived fit.
Does repository context really change which tool an agent picks? Yes. With the same ask on four repositories in four languages, four different email providers won — Resend on TypeScript, Sendgrid on Python, Postmark on Go, Azure ACS on Java. Modifying repository context changed outcomes entirely.
What should operators instrument first? Decision telemetry: which alternatives the agent considered, what evidence it gathered, and which sources it consulted. Consensus across agents is only 42%, so the reasoning behind a pick matters more than the pick itself.