Your Grafana Dashboard Is Lying: 59 of 59 Traces Were One Heartbeat
I thought observability was working. 59 of 59 Tempo traces were the same 250ms cron heartbeat. That was the moment I realized our Grafana dashboard was lying—not with wrong numbers, but with meaningless ones. For a founder/operator, this is a silent killer: you believe you have visibility, but you're flying blind. The stack was Tempo, Grafana, and OpenTelemetry, with three dashboards: Bailian Team Overview, Paperclip Swarm Observability, and Tacavar Ops. Panels rendered, heatmaps looked tight, bar charts were populated, traces flowed. Then we probed the actual query results. The truth was ugly.
The dashboard looked healthy and had exactly one operation
Every panel on Paperclip Swarm Observability showed activity. Latency heatmaps had color. Bar charts had counts. The trace volume panel showed a steady stream. It looked like a system doing real work. But when we ran the underlying Tempo query for the last hour, we got 59 traces. All 59 were the same operation: paperclip_handle_heartbeat, the 60-second cron polling its own work queue. Zero agent runs. Zero LLM calls. Zero tool calls. Zero task outcomes. The dashboard looked healthy and had exactly one operation. That operation was a heartbeat. A cron job checking if there was work to do—and finding none—was the only thing our observability stack could see. The beautiful heatmaps were just 250ms of consistent, meaningless pulse. This is the empty dashboard trap in disguise: not empty panels, but panels filled with noise that looks like signal.
Why 59 of 59 traces were heartbeat cron noise
The mechanism is simple and devastating. The cron job emitted a trace every 60 seconds. Over an hour, that's 60 traces—close enough to 59 after a restart or timing drift. Because nothing else was instrumented or running, the heartbeat dominated trace volume completely. It had low, stable latency (250ms), so the heatmap looked healthy—tight distribution, no outliers. The bar charts counted those heartbeats as operations. But tempo trace debugging should reveal what your system actually does. Instead, it revealed what your system does when it does nothing. The heartbeat was the only operation because the real work—agent runs, LLM calls, tool invocations—either never happened or was never traced. In a founder/operator context, this is a grafana observability anti-pattern: mistaking activity for usefulness. A cron heartbeat is not a business process. It is a metronome. And you cannot debug a metronome.
Querying Prometheus metrics that never existed
While Tempo was drowning in heartbeat noise, Bailian Team Overview was querying Prometheus metrics like agent_calls_total, agent_tokens_total, and agent_cost_usd_total. These metrics did not exist. They were never emitted. But Grafana didn't throw an error. It gracefully rendered empty styled timeseries—lines that looked flat at zero, indistinguishable from a system that is simply idle. This is the empty dashboard trap: a panel with no data looks like zero activity, not a missing metric. Because the default Grafana behavior is to show "No data" or an empty chart, not to fail loudly. You see a flat line and think, "No calls, no tokens, no cost—everything is quiet." In reality, the metric is not being recorded at all. An opentelemetry dashboard audit would have caught this in seconds: run the query, count the rows, check if the metric exists. But without that audit, you're looking at a lie that looks like truth.
Graceful degradation is an observability anti-feature
In most software, graceful degradation is a virtue. In observability, it is an anti-feature. A broken dashboard that shows an error makes you investigate. A beautifully empty dashboard makes you think your system is working. Missing data looks identical to healthy data unless you probe the underlying queries directly. When Prometheus returns no data for a non-existent metric, Grafana renders an empty panel. When Tempo returns only heartbeat traces, Grafana renders a perfectly normal heatmap. The system degrades gracefully—and you degrade into ignorance. For founders and operators, this is unacceptable. You need observability that fails loudly when data is missing, not one that hides failures behind polished visuals. Tacavar Ops is built on this principle: rendered output is not evidence. Evidence is the raw query result. If you can't count the rows, you don't know anything.
A 5-minute probe: count rows, compare cardinality, inspect spans
You don't need a week-long migration to catch these lies. You need a five-minute probe. First, count rows. For every panel on every dashboard, run the underlying query manually. If it returns zero rows, that panel is lying to you. Second, compare cardinality. How many distinct operations are in your trace data? If the answer is one, you have a problem. In our case, cardinality was one: paperclip_handle_heartbeat. Expected cardinality for a swarm system should be dozens—agent runs, tool calls, LLM completions. Third, inspect spans. Open a few traces and look at the actual spans. Are they all the same? Do they contain meaningful attributes like model name, token count, tool name? Or are they empty heartbeats? For Tempo, this means clicking into a trace and reading the operation names and tags. For Prometheus, it means querying the metric directly and seeing if the series exists. This probe takes five minutes per dashboard. It will save you months of false confidence. Run it on your Bailian Team Overview, your Paperclip Swarm Observability, your Tacavar Ops dashboards. Run it now.
The Tacavar Ops rule: rendered output is not evidence
The rule is simple and absolute: rendered output is not evidence. A heatmap is not evidence. A bar chart is not evidence. A green panel is not evidence. Evidence is the raw query result—the rows, the spans, the metric series. If you cannot count the rows, you do not know if your system is working. If you cannot list the operations, you do not know if your traces are meaningful. If you cannot verify the metric exists, you do not know if your dashboard is telling the truth. This is the core of Tacavar Ops. We built it because we fell into the empty dashboard trap ourselves. We saw beautiful dashboards and assumed they were working. They were lying. Now we probe everything. We count rows. We compare cardinality. We inspect spans. We treat every rendered output as a hypothesis, not a fact. You should do the same. Your Grafana dashboard is probably lying to you right now. The only way to know is to ask it for the raw data. Audit your observability stack with Tacavar Ops at tacavar.com/ops.