My Grafana Dashboard Looked Perfect — But Showed Zero Real Data
My observability dashboard looked perfect. Then I probed it: 59/59 traces were a 250ms heartbeat.
I had stood up a Tempo + Grafana + OpenTelemetry stack with three dashboards: Bailian Team Overview, Paperclip Swarm Observability, and Tacavar Ops. Panels were rendering, latency heatmaps were tight, bar charts were populated, traces were flowing. It looked like a textbook observability setup. But when I dug into the actual query results, I found that 59 of 59 traces in the last hour were the same single operation: paperclip_handle_heartbeat — a 60-second cron polling its own work queue. Zero agent runs, zero LLM calls, zero tool calls, zero task outcomes. Meanwhile, the Bailian Team Overview dashboard queried Prometheus metrics (agent_calls_total, agent_tokens_total, agent_cost_usd_total) that don't exist at all. The panels gracefully rendered as empty styled timeseries, indistinguishable from 'low but real' data.
The Beautiful Dashboard That Lied
This is the classic dashboard anti-pattern: aesthetic perfection masking data absence. We optimize for visual polish — consistent color palettes, aligned axes, smooth gradients — and forget that the primary job of a dashboard is to surface truth, not beauty. In my case, the Grafana panels were so well-designed that missing data looked like healthy, low-volume activity. The heatmap showed a single consistent latency band, the bar chart had one bar per minute, and the trace list was populated. Everything looked operational. But it was a mirage. The system was doing nothing except checking its own pulse.
How Missing Data Masquerades as Healthy Signals
Graceful degradation is an anti-feature for observability. When a Prometheus metric doesn't exist, Grafana doesn't show an error — it shows an empty panel with a flat line at zero. When OpenTelemetry spans are all from the same heartbeat, Tempo still renders a beautiful waterfall. The visual system treats absence as data: zero is a valid value, a single operation is a valid pattern. This is especially dangerous for founders and operators who rely on dashboards for quick health checks. A broken dashboard makes people investigate; a beautifully empty one makes people think their system is working. The signal-to-noise ratio collapses because the noise looks exactly like signal.
Probing the Query: The Only Way to Trust a Dashboard
I learned the hard way: never trust the rendered output alone. Always run the query and count the rows. In my Tempo data source, I ran a simple { } | count() and saw 59 traces — all from the same operation. In Prometheus, I queried agent_calls_total and got no data. That's the only way to distinguish between 'zero real activity' and 'missing metric'. For any monitoring stack, build a habit of probing the raw query results before believing the dashboard. This is especially critical when using OpenTelemetry, where instrumentation can be partial or misconfigured. A dashboard that looks perfect but has no real data is worse than a broken one — it gives false confidence.
Three Independent Health Dimensions for Observability
To avoid the empty-dashboard trap, I now design observability around three independent health dimensions:
- Volume: Are traces, metrics, and logs arriving at expected rates? Use a baseline of total events per time window. If the count drops below a threshold, alert — even if the data looks clean.
- Diversity: Are multiple distinct operations present? A single operation (like a heartbeat) is a red flag. Track cardinality of operation names, endpoints, or error types.
- Freshness: Is data recent? A dashboard that shows data from 10 minutes ago might look fine, but if the pipeline stalled, you're flying blind. Alert on staleness.
These dimensions apply regardless of stack — Grafana, Tempo, Datadog, or custom. They force you to look at the meta-health of your observability, not just the surface metrics.
Designing Alert Logic for Truth, Not Looks
Alert logic must be built on these dimensions, not on dashboard aesthetics. For example, instead of alerting on 'latency > 500ms', alert on 'trace count < expected minimum' or 'unique operation count < 2'. In my Tacavar stack, I now have an alert that fires if the number of distinct trace names in the last 5 minutes drops below a threshold. That catches the heartbeat-only scenario immediately. Similarly, I alert on Prometheus metric absence: if agent_calls_total has no data for 2 minutes, that's a critical alert. This shifts the focus from 'is the dashboard pretty?' to 'is the data real?'.
Lessons for Any Monitoring Stack
The lessons apply beyond Grafana and Tempo. Any monitoring tool that renders empty data as a flat line or a single-operation trace as a healthy pattern is vulnerable. OpenTelemetry gives you rich data, but it also gives you the rope to hang yourself with partial instrumentation. The antidote is skepticism: probe queries, count rows, check cardinality, and alert on the health of your observability pipeline itself. Don't let a beautiful dashboard lull you into thinking your system is working when it's just running its own heartbeat.
Check out Tacavar's monitoring patterns for real signal detection at tacavar.com.