Agent Telemetry 2026: What Actually Matters for Observability

Most AI agent observability stacks are built on the wrong foundation. They collect raw LLM calls, token counts, and response times — operational details that tell you almost nothing about whether your agent is actually working. At Tacavar, we learned this the hard way: our early telemetry system captured millions of data points but gave zero insight into why agents failed. We rebuilt observability around what actually matters: action audit trails, cost attribution, latency percentiles, and a structured failure taxonomy. Here is what we learned about building agent telemetry that provides signal, not noise.

The Problem with LLM-Centric Observability

The default approach to agent monitoring is to treat every LLM call as the unit of analysis. Tools log prompts, completions, token counts, and latency. This makes sense for debugging individual model interactions, but agents are not just models — they are systems that make decisions, call tools, and execute workflows over multiple turns. Observability at the LLM level misses everything that makes an agent an agent.

Tacavar's first telemetry system suffered from exactly this problem. We could see every prompt to every model, we could calculate per-token costs with high precision, and we could plot response time distributions. But when an agent failed, none of this data explained why. Did it choose the wrong tool? Did it misinterpret tool output? Did it hallucinate a constraint? Did it get stuck in a loop? The LLM telemetry did not answer these questions. It just told us that a model was called, with no context about the decision path that led to that call.

The fundamental issue is scope mismatch. LLM telemetry operates at the wrong layer of abstraction. Agents execute actions, not just LLM calls. Observability should track what agents do, not just what models say.

Action Audit Trails: The Foundation of Agent Observability

Tacavar shifted from LLM-centric logging to action-centric logging. Every agent execution now produces an audit trail that records the sequence of actions taken: tool calls, decisions, external API requests, and state changes. Each action is logged with context, not just raw data.

An audit trail entry includes: the action type (tool call, decision, API request, state update), the inputs and parameters, the result or output, the timestamp, and the trace ID linking it to the broader agent execution. Critically, we also log the rationale — why the agent chose this action. This is captured as a structured field, not unstructured text, enabling downstream analysis.

Action audit trails solve the core observability problem: they provide a complete, ordered record of what the agent actually did. When an agent fails, we trace the audit trail to identify the exact action or decision that led to failure. This is debugging at the agent level, not the model level. We do not need to inspect individual LLM calls unless the audit trail points to a specific model interaction as the culprit.

The audit trail also becomes the foundation for other observability features. Cost attribution aggregates costs at the action level, not the token level. Latency analysis measures end-to-end action duration, not just model response time. Failure detection flags actions that error or return unexpected results. Everything builds on the action audit trail.

Cost Attribution: From Tokens to Actions

Token-level cost tracking is precise but operationally useless. Knowing that an agent consumed 12,345 tokens costs $0.47 does not tell you whether that expenditure was valuable. Tacavar shifted to action-level cost attribution: we calculate the cost of each action type, then aggregate by agent, workflow, and business outcome.

For example, our code review agent's cost attribution shows that LLM calls account for 60% of total cost, tool calls to Git for 30%, and API calls to external services for 10%. Within LLM costs, we can see that architectural analysis actions are the most expensive, while simple refactoring actions are cheap. This granularity lets us optimize: we constrain the expensive analysis actions to use smaller models, while allowing the cheaper refactoring actions to use larger models.

Action-level cost attribution also enables business-level ROI analysis. We can calculate the cost per successful PR, the cost per bug detected, or the cost per unit of code reviewed. Token counts cannot provide this insight. Only by tying costs to actions and outcomes can we determine whether an agent is economically justified.

Latency Percentiles: Beyond Average Response Time

Average latency is a dangerous metric. A single long-running task can skew the average, hiding performance issues in the common case. Tacavar tracks latency percentiles: p50, p90, p95, and p99. We measure latency at the action level and the agent-execution level.

At the action level, p50 tells us the typical duration of each action type. p95 tells us the worst-case experience for 95% of actions. If p95 is significantly higher than p50, we have a latency outlier problem that needs investigation. At the agent-execution level, we measure end-to-end workflow latency, which includes all actions, model calls, and external API requests.

Percentile-based latency analysis revealed performance bottlenecks that averages completely missed. Our code review agent had a p95 latency of 45 seconds, but the average was only 12 seconds. Investigation showed that a small subset of reviews triggered external API calls to security scanners that could take up to 60 seconds. The average hid this problem; p95 surfaced it immediately.

Failure Taxonomy: Structured Error Classification

Failures are not monolithic. A tool timeout is different from a model hallucination, which is different from a permission error, which is different from a logic error. Tacavar developed a structured failure taxonomy that classifies failures by type, severity, and recoverability.

Our taxonomy includes: tool failures (timeout, rate limit, permission denied), model failures (hallucination, refusal, formatting error), logic failures (incorrect decision path, infinite loop, constraint violation), and external failures (API down, invalid response, network error). Each failure type is assigned a severity level: critical, high, medium, or low. Recoverability indicates whether the agent can self-correct or requires human intervention.

This taxonomy enables automated alerting and routing. Critical failures trigger immediate pager alerts. High failures are routed to the on-call engineer. Medium and low failures are aggregated for daily review. Failures that are recoverable trigger automated retry logic. Failures that require human intervention create tickets in our issue tracker.

The taxonomy also powers failure trend analysis. We can see that tool timeouts are increasing, prompting investigation into external service reliability. We can see that model refusals spike after a prompt change, prompting a rollback. Structured classification turns raw failures into actionable insights.

What We Stopped Tracking

Building better observability meant stopping the collection of noise. Tacavar no longer logs raw LLM prompts and completions by default. We only capture them on-demand for debugging specific failures. We stopped tracking token-level costs in production dashboards; that data is now available on-demand, not in our primary observability views. We stopped measuring average response time entirely.

We also stopped tracking vanity metrics. We do not measure "agent happiness scores" or subjective quality ratings. We do not count the number of agents deployed or the number of workflows executed. These metrics do not correlate with business value and create false confidence. We focus on outcomes: success rate, cost per successful outcome, latency percentiles, and failure taxonomy distribution.

The Tacavar Observability Stack in Practice

Our current observability stack has four layers. The data layer ingests action audit trails, costs, latencies, and failures into a time-series database. The analysis layer computes percentiles, aggregates costs, and classifies failures using our taxonomy. The alerting layer routes critical and high-severity failures to the appropriate responders. The dashboard layer presents the signal that matters: success rate by agent, cost per outcome, p95 latency, and failure type distribution.

This stack is purpose-built for agents, not adapted from traditional application monitoring. We do not use APM tools that were designed for HTTP services. We do not use log aggregation that was built for structured text logs. We built a telemetry system that understands agents as first-class entities, with observability primitives that map to how agents actually work.

How to Apply This in Your Own Agent Stack

Start by defining your audit trail format. What is the unit of analysis for your agents? Tool calls? Decisions? API requests? Define a schema that captures action type, inputs, outputs, rationale, timestamp, and trace ID. Implement this as the foundation of your logging system.

Next, build cost attribution at the action level. Map each action to its cost components: LLM calls, API requests, compute resources. Aggregate by agent and workflow, then tie to business outcomes.

Implement latency tracking with percentiles. Do not rely on averages. Measure both action-level latency and end-to-end workflow latency.

Develop a failure taxonomy for your domain. Classify failures by type, severity, and recoverability. Use this taxonomy for automated alerting and routing.

Finally, stop collecting noise. Audit your current telemetry and eliminate anything that does not directly contribute to understanding agent behavior, cost, performance, or failure. Less data, more signal.

Tacavar's agent observability stack is available as part of our AI ops platform. See how action-centric telemetry transforms agent debugging at tacavar.com.