My Grafana Dashboard Was Beautiful and Completely Useless
I had a beautiful Grafana stack. 59 of 59 traces were the same heartbeat cron. Zero actual work.
If you're a founder or operator, you know the ritual: stand up Tempo, wire in Grafana, sprinkle OpenTelemetry traces, and build dashboards that make your infrastructure look like a spaceship cockpit. I did exactly that for the Tacavar platform. Three dashboards, sleek panels, latency heatmaps, and a steady stream of traces flowing by. The observability stack looked like a model citizen. It was a lie wearing a pretty face.
The Setup: Tempo, Grafana, OpenTelemetry, and Three Dashboards
Tacavar's internal monitoring stack was textbook: Tempo for tracing, Prometheus for metrics, Grafana for visualization, and OpenTelemetry instrumentation across services. On paper, it was the modern SRE dream. I built three dashboards: one for the Bailian team overview, one for the Paperclip swarm, and one for Tacavar's core operations. Each had panels for request rates, error budgets, latency percentiles, and resource saturation. All the right panels, all the right queries, all the right tooling. The problem wasn't the technology — it was that I trusted rendered output instead of probing the underlying data. That trust turned a healthy-looking UI into an observability anti-pattern machine.
What the Panels Showed Me
Open the dashboards and the story was immaculate. The latency heatmap for the Paperclip swarm showed tight, sub-300ms bands. The bar charts for agent activity were populated with clean, periodic spikes. The trace waterfall views in Grafana were flowing — spans appearing every second, each with proper parent-child relationships. If I'd taken a screenshot and posted it on social media, it would have earned admiring nods from other founders. The panels suggested a system that was busy, healthy, and humming. But here's the uncomfortable truth about dashboards: they are only as truthful as the queries behind them. And I hadn't run those queries. I'd just watched the pixels render.
Probing the Queries: 59 of 59 Traces Were a Heartbeat
One day I finally did what I should have done on day one: I clicked into the trace explorer and asked for every trace in the last hour. What came back was a wall of identical operations. 59 of 59 traces in that hour were paperclip_handle_heartbeat — a 250ms cron job that polls its own work queue and finds nothing. That's not a system at work; that's a system checking its own pulse. Zero agent runs. Zero LLM calls. Zero tool calls. Zero task outcomes. Every single trace in my beloved Tempo store was a literal heartbeat. The histogram looked like a healthy service because a heartbeat is technically an operation. But ops that don't do work are not observability signals — they're noise dressed as telemetry. The reason this happens is that we instrument what we know how to instrument, not what matters. The heartbeat was an easy span to emit. The real work — asynchronous agent loops, tool executions, cost accumulation — wasn't being traced at all. The instrumentation was correctly reporting that a piece of code ran. It was completely silent about whether the business was moving.
The Missing Metrics That Rendered as Empty Styled Timeseries
While I was digging, I checked the Bailian Team Overview dashboard. It queried Prometheus metrics named agent_calls_total, agent_tokens_total, and agent_cost_usd_total. Those metrics don't exist anywhere in the Prometheus endpoint. No exporter publishes them. No service emits them. There is zero data behind those panels. Yet the dashboard didn't break. It didn't show a red warning or a NO_DATA error. Instead, Grafana gracefully rendered empty timeseries as styled, axis-labeled, legend-bearing panels that looked like a low but real traffic pattern. A visitor would have to squint to distinguish between a panel showing 0.001 requests per second and a panel showing absolutely nothing. This is one of the nastiest observability anti-patterns in modern infrastructure: graceful degradation that converts missing data into a friendly, blank-ish chart. If your panel can't tell the difference between zero and nothing, your dashboard is not a monitoring tool — it's a lullaby.
Why Graceful Degradation Is an Anti-Feature
In most software, graceful degradation is a feature. A website that returns a cached card when the API is down is better than a white page. But observability is the one domain where graceful degradation is sabotage. A broken dashboard makes people investigate — they see a missing $ variable, a template error, a red panel, and they say "something's wrong." That triggers investigation, which surfaces the actual absence of data. A beautifully empty dashboard does the opposite: it makes people think the system is fine. It lulls you into allocating infrastructure budget to a monitor that is watching nothing. The Tacavar stack had no red alerts, no broken query templates, no obvious failure. It was the most dangerous state an operator can be in: confidently blind. Graceful degradation here means the failure mode is invisible. The absence of data is indistinguishable from the presence of low activity. For agent monitoring specifically, this is fatal. An AI agent either completes tasks or it doesn't. There is no "low but healthy" ground state. If your dashboard can't show a hard zero as a hard zero, you don't actually know whether your agents are working or dead.
How to Audit Your Own Observability Stack
Here's the audit I now run on any observability stack before calling it done — and the one you should run this week. First, for every panel on your Grafana dashboard, run the underlying query manually in the Prometheus or Tempo query UI. Don't look at the rendered panel; look at the raw result. If the result is empty, you have a dead panel, not a quiet panel. Second, count the distinct kinds of traces. Don't just check that traces are flowing; list the operation names. If you see more than 50% of your traces coming from health checks, heartbeats, or polling loops, your agent monitoring is a pulse monitor, not a work tracker. Third, instrument outcomes, not just requests. paperclip_handle_heartbeat is an outcome-neutral span. What you care about is paperclip_task_completed, paperclip_task_failed, paperclip_llm_call, paperclip_tool_used. If one of those doesn't appear as an OpenTelemetry trace in the last hour, your system is not working — regardless of how many heartbeats succeeded. Fourth, write a simple script that counts rows per operation per hour and alerts if any business-critical trace type goes to zero. This is the single highest-leverage thing you can add to any Tempo + Grafana setup. Finally, treat every dashboard as a liability until you've proven its queries return real, differentiated, high-cardinality data. If a panel can't survive a missing metric without lying, replace it with a text box that says "UNKNOWN." That would be more honest.
The Generalizable Lesson: Count Rows, Not Panels
The lesson from the Tacavar observability audit is embarrassingly simple: never trust the rendered output alone. Always run the query and count the rows. A Grafana dashboard with 50 beautiful panels is worth less than one query that shows agent_llm_calls_total has a non-zero count. The obsession with dashboard aesthetics — gradients, heatmaps, sparklines — is a placeholder for actual operational rigor. When you're a founder or operator, your time is too scarce to spend it admiring charts. The only acceptable use of observability is to answer three questions: Are we doing work? What work are we doing? Is the work succeeding? Anything else is decoration. The 59-of-59 heartbeat incident taught me that the worst outage at Tacavar was never a crash — it was a system that was perfectly alive and completely useless. Don't let your stack fool you the same way. Put your own infrastructure under Tacavar's monitoring microscope at tacavar.com.