My Observability Stack Looked Perfect. It Was Showing Nothing.
The dashboards looked gorgeous. All 59 traces in the last hour were the same cron heartbeat. I had just stood up a Tempo + Grafana + OpenTelemetry stack for Tacavar, and every panel rendered like a data-center ad: tight latency heatmaps, populated bar charts, traces flowing in with the rhythm of a trading floor. I was proud. I was wrong.
I didn't catch it by looking at the dashboards. I caught it by running the actual query and counting the rows. That's when the whole beautiful picture collapsed.
The Dashboard That Looked Too Good to Be True
The stack was textbook. Tempo for traces, Grafana for visualization, OpenTelemetry agents running alongside every Tacavar service. We built three dashboards: Bailian Team Overview, Paperclip Swarm Observability, and Tacavar Ops. Each one was laid out with the care of a product launch.
The latency heatmaps were tight. The bar charts were populated. The trace stream never went silent. From a visual standpoint, the system was perfect. I remember thinking: this is what observability is supposed to look like.
Then I probed the query results. Not the rendered panels — the actual query results underneath. And the numbers didn't match the story.
Why 59 of 59 Traces Were One Heartbeat
I ran a simple range query against Tempo for the last hour. The result: 59 traces. That's roughly a trace per minute, which looked healthy on the panel. But when I expanded the operation names, all 59 were the same single operation: paperclip_handle_heartbeat, the 60-second cron that was polling its own work queue.
Zero agent runs. Zero LLM calls. Zero tool calls. Zero task outcomes. The entire observable behavior of the system was a self-ping.
This is the classic OpenTelemetry agent monitoring anti-pattern: we instrumented the container, the process, the agent itself — but not the actual work the system was paid to do. The heartbeat cron generated a trace every 60 seconds, so the trace stream never went idle. The heatmaps showed healthy p95s because the heartbeat always completed quickly. And the panels rewarded us with the reassuring shape of a busy system.
But the system was doing nothing. If you only look at trace volume, you'll confuse heartbeat traffic with real work. A clock ticking is not a business. You have to ask: what operations are actually making up the trace stream? If the answer is one internal poller, you're not observing your system; you're watching a metronome.
The Missing Metrics That Rendered as Empty Beauty
The trace data was useless, but I held out hope for the Prometheus metrics. The Bailian Team Overview dashboard queried agent_calls_total, agent_tokens_total, and agent_cost_usd_total — the exact metrics that power every agent platform pitch.
Only one problem. Those metrics don't exist anywhere in the stack. No exporter emitted them. No service ever wrote to those series. But Grafana rendered the panels anyway.
That's the grafana dashboard empty no data trap. A panel expecting a series that doesn't exist will gracefully render as an empty styled timeseries. If the panel has thresholds or non-zero min/max from a stale cache, it can look just like a low-but-real signal. Your brain interprets "empty line on a graph" as "no activity," not "no data." But there's a huge difference.
An empty timeseries is not a signal of zero; it's a signal of nothing. And Grafana doesn't tell you which one you're looking at.
I remember staring at the cost panel. It showed a flat line near zero with a faint tick. In my mind, that meant the system was cheap to run — which I had bragged about in a standup. In reality, the series had no data at all. The panel was not reporting a cost of zero; it was reporting an absence of measurement.
Metrics that don't exist are worse than metrics that are wrong. Wrong metrics can be debugged. Nonexistent metrics are silently treated as "empty but fine."
Graceful Degradation Is an Anti-Feature
The software engineering instinct is to make dashboards degrade gracefully. If a query returns no data, don't throw an error — show an empty panel. But for observability, graceful degradation is an anti-feature.
A broken dashboard makes people investigate. A dashboard that shows a beautiful empty panel makes people think the system is working. The reason is simple: missing data looks identical to healthy data unless you probe the underlying queries directly.
Think about it. If a panel that should show request latency shows nothing, is that a good thing or a bad thing? In most organizations, that panel being empty reads as "no traffic" — which is plausible. It takes a disciplined operator to ask: is there actually no traffic, or is the telemetry pipeline broken?
I've seen teams run for weeks on empty dashboards because the absence of data was interpreted as the absence of activity. That's the trap. The dashboards didn't fail; they lied — politely, professionally, with beautifully styled empty lines.
That's why the concept of "trust" is so important. You build observability dashboard trust by verifying the underlying queries, not by admiring the pixels. The first time you catch your dashboard lying, the trust is gone. The only way to earn it back is to change what you look at: not the rendered output, but the query results themselves.
How to Probe Your Own Observability Stack
You don't need any special tool to avoid this trap. You just need a simple, repeatable audit you run on every dashboard you own.
Start with traces. Open your tracing backend — in my case, Tempo — and run a range query that lists all operation names instead of just the total count. Don't accept "59 traces" as an answer. Break it down by operation. If 59 of 59 are the same heartbeat, you have a tempo trace observability audit to do.
Next, metrics. For every Prometheus metric referenced in your dashboards, run a query that returns the series list. I used this exact command in PromQL: {__name__=~"agent_calls_total|agent_tokens_total|agent_cost_usd_total"}. The result was empty. Not zero, not stale — empty. And that's the difference. count(<expression>) will tell you if the series exists, even if the value is zero. If the series doesn't exist, you get 0 from the count, but the panel still renders as if it's a flat zero.
Now do the same for every metric. Check cardinality. Is the series written to only when something happens, or is it monotonically increasing with a timestamp? A good test: query the last 24 hours of raw samples and confirm you see timestamps. If there are no samples, the metric is not being emitted.
Set an alert on "no data" for your most critical panels. Most teams alert on high latency or error rates. Alert on the opposite: if a metric you depend on isn't arriving at all. That alert will fire far more often than you want, and that's the point.
Finally, run a query-level audit on a schedule. Don't just build dashboards and forget them. Every week, pick one panel and run the query manually. Count the rows. Look at the raw result. Verify that the volume of data matches what you'd expect. This takes ten minutes and will catch the majority of "pretty but empty" dashboards before they ruin your trust.
The Rule I Now Use Before Trusting Any Dashboard
I've learned one rule that I now use before trusting any dashboard, and it's dead simple: never trust the rendered output alone. Always run the query and count the rows.
If you see a panel with a flat line, run the query. If you see a trace stream, list the operations. If you see a cost metric, check that the series exists. The rendered panel is the dashboard's opinion. The query result is the ground truth.
At Tacavar, we now bake this into our observability workflow. Every dashboard in the Tempo/Grafana stack is paired with a "health of the health check" — a separate monitoring query that watches the telemetry itself. If the underlying data sources stop emitting, we want to know within minutes, not discover it months later by accident.
The empty panel trap is not a niche problem. It's the default state of most dashboards I've seen in the field. Founders and operators get lulled into trusting a dashboard because it's beautiful, and then make decisions about scaling, cost, and agent performance based on data that never existed.
The fix is uncomfortable because it forces you to audit yourself. But the alternative is worse: running a system where the only thing you can actually observe is your own heartbeat.
Run the same query-level audit on your stack at tacavar.com/observability-audit.