My Grafana Stack Was Beautiful and Watching Exactly One Cron Job
Three Grafana dashboards, 59 identical heartbeat traces, and zero real signal. How Tacavar audits empty dashboards, dead alerts, and ai agent stack observability gaps.
59 of 59 traces were the same heartbeat. My dashboards looked perfect anyway.
If you run infrastructure, you've probably built this exact stack: Tempo, Grafana, OpenTelemetry, a few dashboards with latency heatmaps and bar charts that actually render. Ours covered the Tacavar agent operations — three dashboards, panels populated, traces flowing, the whole thing visually indistinguishable from a monitoring system doing its job. Then I ran the underlying queries by hand and found that everything I was looking at was a lie of omission. This is the story of how an AI agent stack can look fully observable while telling you nothing, and the audit process Tacavar now uses to prevent it.
Three Dashboards, One Operation, Zero Visibility
The stack was real. Tempo ingesting spans, Grafana rendering panels, OpenTelemetry instrumenting the services. Three dashboards: a team overview, a swarm-level view of our paperclip-generation agents, and a Tacavar Ops board. Heatmaps tight, bar charts populated, traces arriving by the minute.
The problem: when I actually probed the trace store, 59 of 59 traces in the last hour were the same single operation — paperclip_handle_heartbeat, a 60-second cron polling its own work queue. Zero agent runs. Zero LLM calls. Zero tool invocations. Zero task outcomes. Every panel on the swarm observability dashboard was drawing its data from one 250ms heartbeat loop, and the visualization layer was perfectly happy to present that as "healthy traffic."
This is the first of the grafana observability anti-patterns worth naming: a dashboard measures what flows through it, not what your system does. If your only instrumented path is a self-polling cron, your beautiful dashboard is a monitor for the cron and nothing else. The agents — the actual product — were invisible.
Graceful Degradation Is an Anti-Feature in Observability
The team overview dashboard was worse, in a subtler way. It queried Prometheus metrics — agent_calls_total, agent_tokens_total, agent_cost_usd_total — that didn't exist. Not low. Not sparse. Absent from the metrics registry entirely. Grafana's response was to render empty, styled timeseries panels that were visually indistinguishable from "low but real" data. Nothing errored. Nothing flagged. The dashboard degraded gracefully into looking functional.
Here's why that's an anti-feature rather than a convenience: a broken dashboard makes people investigate. A beautifully empty one makes people conclude the system is working. In empty dashboard monitoring, missing data and healthy data render identically unless someone deliberately probes beneath the rendered output. That asymmetry is dangerous, because the person most likely to trust the panel at a glance is the one least likely to run the raw query — usually whoever's on call at 2 a.m.
Tacavar's position now is explicit: observability components should fail loudly. A panel with no underlying series should tell you so in the panel itself, not through subtle negative space that reads as calm.
Probing the Queries: Count Rows, Not Panels
The fix is procedural, not architectural. For every dashboard, run each panel's underlying query directly and count what comes back. Not "does the panel render" — "how many rows does this query return, and are those rows the operations I think I'm monitoring?"
For the trace data, that meant an opentelemetry trace analysis pass against Tempo: pull the last hour of spans, group by operation name, and look at the distribution. In our case the distribution was a single value at 100%. For the metrics, it meant querying Prometheus for the metric names themselves and confirming they existed. Three didn't. Those dashboards had been up for weeks rendering confident-looking emptiness.
The discipline: rendered output is evidence of the visualization layer, not of your system. Only row counts against the source of truth constitute evidence about your system. Any ai agent stack observability setup that hasn't been probed this way is a hypothesis, not a monitoring capability.
Alerting on Metrics That Don't Exist
There's a second-order failure most teams miss: alerting rules bound to nonexistent metrics never fire, and never tell you they can't fire. If you have an alert on agent_cost_usd_total and that metric was never emitted, your cost-spike alert isn't "quiet because costs are fine." It's dead, silently, indefinitely.
Audit every alert rule against the actual metric registry. Every rule referencing a metric name should resolve to something being actively scraped. At Tacavar we now treat a rule whose target metric has zero reported series as an alert condition in itself — a watchdog on the watchdogs. The same logic applies to trace-based alerts: if your "agent run latency" alert keys on an operation name that hasn't appeared in 24 hours, that's not a healthy quiet period, it's instrumentation loss, and it should page someone.
The Heartbeat Trap: High Volume Is Not High Signal
The heartbeat deserves its own section because it's the most seductive failure mode. paperclip_handle_heartbeat ran every 60 seconds, reliably, producing a steady stream of spans. Trace volume looked strong. Latency was a tight, satisfying 250ms. Every heatmap cell glowed the right color. By every volume-based signal, the system appeared alive.
It wasn't. The heartbeat proved only that the cron scheduler and the work queue were reachable by each other. It carried no information about agent runs, model calls, tool executions, or task completion — the operations that constitute the product. High volume is not high signal. A cron heartbeat is the observability equivalent of a pulse: it confirms the patient isn't dead and nothing more.
The countermeasure is cardinality-of-operations auditing. For any window, enumerate distinct operation names in your trace store and check them against the operations your architecture actually performs. If the set of observed operations is a strict subset of the set of expected operations — if heartbeats, health checks, and queue polls are the only entries — your instrumentation is coverage-gapped regardless of how much traffic those traces represent. Tacavar's ops tooling runs this check continuously and surfaces the delta between observed and expected operations as the primary health signal, not trace volume.
A 10-Minute Audit Checklist for Your Own Stack
Run this against any observability stack, including your own:
- List your dashboards, then list the operations your system performs. For each dashboard, name the specific business operations it should be showing you. If you can't, the dashboard is decorative.
- Run every panel's query and count rows. Rendered panels prove nothing. Row counts against Tempo or Prometheus prove the data exists.
- Group traces by operation name over the last hour. If one operation dominates — especially a heartbeat or health check — you're watching infrastructure self-talk, not your product.
- Verify every queried metric exists. Query the metric names directly. Nonexistent metrics should be treated as incidents, not gaps to fix later.
- Audit every alert rule's target. A rule on a missing metric is a dead sensor. Replace it or delete it knowingly.
- Compare observed operations to expected operations. Any expected operation absent from your trace store is an instrumentation gap. Track that gap as a metric of its own.
Ten minutes of probing found more truth in our stack than weeks of admiring the dashboards had. The rendered layer will always tell you a flattering story; only the underlying rows can contradict it. Tacavar builds monitoring that tells you when your AI stack is lying to you. See the ops and watchdog capabilities behind it at tacavar.com/ops.