How to Evaluate AI Agent Frameworks: A Production Rubric
A seven-dimension scorecard for evaluating AI agent frameworks before production: reliability, latency, cost per task, observability, security, vendor lock-in, and developer velocity.
Every team picking an AI agent framework in 2026 runs the same search: LangGraph vs CrewAI vs AutoGen, three benchmark blog posts, two Reddit threads, and a demo that works beautifully on the hello-world task. Then the framework hits production, and the questions that matter — what happens when the model returns malformed JSON, how much a completed task costs, who can see the traces — were never asked. This page is the rubric we wish existed when we deployed our first agents. It is the evaluation checklist behind Tacavar's 12-agent production stack, distilled into seven dimensions you can score any framework against in an afternoon.
The rubric is opinionated on one point: a framework is a bet on a failure model, not a feature list. Demos sell features. Production lives in the failure modes. Every dimension below scores the framework on what it does when things break, because that is where the framework actually earns its place in your stack.
The Seven-Dimension Scorecard
Score each dimension 1-5. Anything below 3 on reliability, cost, or observability is a disqualifier regardless of the total — those three are load-bearing for production. The full rubric with weights:
- Reliability under failure (weight: 25%) — retries, state durability, transactional behavior
- Latency profile (weight: 15%) — p50/p95 task completion, streaming, parallelism
- Cost per task (weight: 20%) — real token burn, model-tier routing support
- Observability (weight: 15%) — traces, token accounting, replayability
- Security and containment (weight: 10%) — tool permissions, sandboxing, secret handling
- Vendor lock-in (weight: 10%) — provider portability, export format, self-host story
- Developer velocity (weight: 5%) — onboarding time, debugging ergonomics
The weights came from post-mortems, not aesthetics. Reliability and cost dominate because those are the two ways an agent stack quietly becomes unrunnable: it fails too often, or it costs too much per failure. Vendor lock-in matters less than people fear — model providers swap faster than frameworks do.
Dimension 1: Reliability Under Failure
Score this dimension by answering three questions with code, not docs. First: when the model returns malformed output mid-task, does the framework retry with repaired context or crash the run? Second: when a process dies mid-graph, can you resume from the last durable checkpoint or must the task restart from zero? Third: when a tool call hangs, does a timeout fire or does the run leak?
LangGraph earns its production reputation here: checkpointing is a first-class primitive, and graph state survives process restarts. CrewAI has improved its retry handling but historically delegated durability to the application layer. AutoGen's conversation-thread model is durable in principle but resume semantics depend heavily on which runtime you pair it with.
The scoring test we run: kill the process at a random point during a 20-step task, 20 times. A framework scores 5 if the task resumes and completes every time. It scores 1 if any kill corrupts state or requires manual cleanup. Most frameworks land at 3 — resumable with configuration you will not have written until the first 2 AM incident.
Dimension 2: Latency Profile
Latency evaluation fails when teams measure token generation speed instead of task completion time. The number users feel is p95 end-to-end task latency: every model call, tool round-trip, retry, and framework overhead summed. A framework that adds 200ms of graph-dispatch overhead per node is invisible in a demo and catastrophic in a 40-node pipeline.
Score by benchmarking a representative task — not hello world — at p50 and p95. Then check two capabilities: does the framework support parallel node execution natively, and does it stream intermediate output so users see progress instead of a spinner? We run the detailed numbers in our framework comparison; the short version is that dispatch overhead varies by nearly 10x between frameworks, and only the p95 numbers expose it.
Dimension 3: Cost Per Task
Per-token pricing is a vendor number. Cost per completed task is your number. The difference is retries, context re-sends, and the model-tier mix the framework makes easy or hard. A framework that natively supports routing — cheap model for classification steps, frontier model for synthesis — will cut your task cost by 5-10x compared to one that assumes a single model per run.
Score this by running 100 representative tasks and dividing total spend (tokens times price, all tiers) by completed tasks. Include failures: a task that retries three times before succeeding costs 3x its nominal price. We publish our full methodology and the real numbers — our stack runs at under $0.10 per task — in the AI agent cost-per-task benchmark. A framework scores 5 on this dimension if token accounting is built in and model routing is native; it scores 2 if you are wrapping the billing API yourself.
Dimension 4: Observability
The first production incident will ask a question your logs cannot answer: which prompt, which tool call, which model decision produced this output? Frameworks differ enormously on whether that question is answerable. Look for three things: span-level traces that correlate every model call to its task, per-node token accounting, and replay — the ability to re-execute a recorded trace against a new prompt or model.
Replay is the most underrated capability in the list. Debugging a failed agent run without replay means reconstructing state by hand; with replay it is a unit test. Our full position on what instrument and what to ignore is in agent telemetry and observability. Scoring: 5 = native OpenTelemetry export with token dimensions; 3 = hook system you can build on; 1 = print statements and prayer.
Dimension 5: Security and Containment
Agents execute model-generated instructions against real tools. The evaluation question is what stops a confused or manipulated model from doing damage: can tool permissions be scoped per-node, are secrets injected at call time or baked into graph config, and is there a first-class mechanism for killing a runaway run? We treat the last one — the circuit breaker — as mandatory, and document the patterns in agent circuit breaker patterns.
Most frameworks score poorly here because security is not a demo feature. Score 5 = per-node permission scopes plus native run termination; score 1 = root-level tool access with no kill switch. Anything scoring 1 requires you to build the containment layer yourself, and you should budget for that as a real project, not a config file.
Dimension 6: Vendor Lock-In
Lock-in evaluation is mostly a portability test: how much of your graph definition, prompts, and evaluation harness survives a model-provider switch, and could you self-host the control plane if a vendor changed pricing or terms? The saving grace of this ecosystem is that switching costs are lower than they look — model providers are more interchangeable than frameworks, and prompt assets are plain text. The dangerous lock-in is not the LLM vendor; it is proprietary state formats and managed-only control planes.
Score 5 = open-source runtime, serializable state you can export, provider-agnostic model interface. Score 1 = managed-only platform with proprietary orchestration DSL. Note the asymmetry: a 1 here is survivable for internal tooling and near-fatal for anything customer-facing.
Dimension 7: Developer Velocity
The lowest-weighted dimension deliberately. Velocity matters for the first month; reliability and cost matter every month after. Still, measure it honestly: time for a competent engineer without framework experience to ship a two-tool agent to staging. LangGraph's learning curve is real — graph semantics take a week to internalize. CrewAI optimizes for fast starts. Neither wins if your team cannot debug what it built, so weight the debugging ergonomics over the onboarding speed.
Scoring the Major Frameworks
Applying the rubric to the three frameworks we get asked about most, from our production experience rather than their docs:
- LangGraph — Reliability 4.5, Latency 4, Cost 4, Observability 4, Security 3.5, Lock-in 5, Velocity 2.5. The production default. Checkpointing and graph durability are real. You pay for it in onboarding time.
- CrewAI — Reliability 3, Latency 3.5, Cost 3.5, Observability 3, Security 3, Lock-in 4, Velocity 4.5. Fastest path from idea to working multi-agent demo. Durability is your job.
- AutoGen (Microsoft) — Reliability 3.5, Latency 3, Cost 3, Observability 3.5, Security 3, Lock-in 3.5, Velocity 3.5. Strong research lineage and conversation-based patterns; runtime choices determine half your score.
These are starting scores from our workloads — yours will differ, which is the point of a rubric over a verdict. For the deeper treatment of where agent evaluation as a discipline is heading, including why benchmark scores and production outcomes diverge, read the agent evaluation inflection arc.
The Downloadable Checklist
The condensed version we run before committing to a framework. Copy it into your evaluation doc:
- Kill the process mid-task 20 times. Score resume behavior.
- Benchmark p50/p95 task latency on a representative workload, not a demo.
- Run 100 tasks; divide total spend by completions. Include retry cost.
- Confirm span-level traces with per-node token accounting exist natively.
- Verify per-node tool permissions and a tested kill switch.
- Export the graph state to disk. Confirm it re-imports.
- Time a cold-start: new engineer, two-tool agent, staging deploy.
- Disqualify on any sub-3 score in reliability, cost, or observability.
What We Would Tell You To Do Differently
Teams over-index on framework choice and under-index on measurement. The honest read after running agents in production: the framework determines maybe 20% of your outcomes; your observability, cost discipline, and containment patterns determine the rest. A mediocre framework with excellent telemetry beats an excellent framework with none, because you can fix what you can see. Pick the framework that lets your team ship, then invest everything you saved from the framework holy war into the boring disciplines: traces, benchmarks, circuit breakers, and a real cost-per-task number.