Replay Your Agent Runs: Diffing JSONL Traces
Dashboards tell you an agent failed. A JSONL trace diff tells you which tool call went wrong. The minimal trace schema, a diff recipe, and an honesty table.
On August 30, 2026, a founder posted on Hacker News: an AI agent misread a risk signal and moved $1.2M in trades. That figure is the founders' own account, not an audited number — but the shape of the failure is familiar to anyone running agents against real work. The agent did something wrong at 2 a.m., the dashboard recorded it, and the next morning the team had a red panel and a question: which decision, in the middle of that run, actually went wrong?
Dashboards answer "did it fail." They do not answer "where." This post is about the tool that does: replaying the run from its JSONL trace and diffing it against a known-good run until the divergence shows up as one changed tool call. A founder recently built exactly this tool for his own agent stack and published the recipe. It is forty lines of code and an afternoon. Most teams with agents in production do not have it.
The distinction matters because guardrails and replay solve different halves of the same problem. Guardrails are policy: they stop the bad call before it executes. Replay is forensics: after the bad call executes anyway, it tells you what to fix. If your guardrails piece covers the control stack, this piece covers the evidence trail.
The one thing to log
Every agent run should be appendable to a single JSONL file — one line per step, in execution order, written before the step's result is used downstream. Log the inputs before the outputs, or a crashed run leaves you with conclusions and no premises. A minimal schema:
{"ts": "2026-08-30T06:14:22Z", "step": 14, "type": "tool_call",
"agent": "risk-monitor", "tool": "get_positions",
"args": {"venue": "binance"}, "latency_ms": 812}
{"ts": "2026-08-30T06:14:23Z", "step": 15, "type": "tool_result",
"agent": "risk-monitor", "tool": "get_positions",
"result_hash": "a3f1…", "result_summary": "14 open positions, 1 flagged"}
{"ts": "2026-08-30T06:14:23Z", "step": 16, "type": "reasoning",
"agent": "risk-monitor",
"content_hash": "9c2e…", "tokens_in": 4210, "tokens_out": 388}
Three fields do most of the work: step (total order), tool + normalized args (what was asked), and a hash or short summary of the result (what came back). Reasoning content gets hashed rather than stored verbatim — you need to know that it changed and when, not a transcript you'll never read. Model name and version go in a header line at the top of the run.
Total payload: a few hundred bytes per step. At 10,000 steps a day you are logging megabytes. There is no excuse not to.
The diff recipe
Replay is a diff, not a video. The procedure, once the traces exist:
- Pick a golden run. The last run that behaved correctly, closest in time and inputs to the failure. Trace diffs rot with distance — a run three versions back diverges for boring reasons.
- Canonicalize before comparing. Normalize the
args— sorted keys, rounded floats, redacted timestamps and request IDs. If you diff raw JSON you will spend the afternoon chasing serialization noise. - Diff in step order. Walk both runs step by step. Most steps match. You are looking for the first divergence: the step where the golden run calls
get_risk_limitsand the failed run callsget_positionsinstead; or where both call the same tool but one passesthreshold=0.05and the otherthreshold=0.5. That first divergence is almost always the call that went wrong. Everything after it is downstream damage. - Annotate the divergence, don't just find it. Record why it happened if the trace shows it — a changed model version, a drifted memory entry, a tool that started returning a different schema. The diff localizes the failure; the annotation explains it. Teams that skip step 4 fix the same bug twice.
Two properties make this cheap. Diffs are between runs, not against a spec — you never have to write the spec. And the first divergence dominates: in practice, the offending call is within the first three divergent steps, and the rest of the diff is noise you can ignore.
Run-level diffing also catches the failure class dashboards structurally miss: silent wrongness. An agent that gets a worse result through a perfectly healthy sequence of calls produces no red panel, no error spike, nothing to alert on. A diff against a golden run surfaces it in one step.
The honesty table
Replay is not observability, and it is not a guardrail. Here is the honest ledger:
| What replay catches | What replay misses |
|---|---|
| The specific tool call that diverged from a good run | Failures where every call was "correct" but the goal was wrong |
| Silent regressions — same task, worse outcome, no errors | Bugs that exist in the golden run too (diffs against a bad baseline) |
| Behavior drift across model or prompt versions | Non-determinism the canonicalizer accidentally erased |
| The exact moment a memory/context change altered behavior | Anything not represented in the trace (unlogged side effects, external state) |
| Reproducible evidence for a post-mortem | Real-time prevention — replay is after the fact by definition |
The last row on the left and the last row on the right are the whole argument for running both halves. Guardrails exist so the $1.2M call never fires. Replay exists so that when something fires anyway, you find the step in an afternoon instead of a quarter. Teams tend to build the first half and wonder why the same incident recurs; the recurring incident is the one nobody diffed.
Wiring it into the stack
Replay becomes valuable the day it is wired into the two systems that already exist. First, the guardrail layer: the step number the diff flags is the step your policy engine should have watched, and the annotation from step 4 of the recipe is the regression test your control plane was missing. Second, the routing layer: if you route across model versions or providers, a trace diff between runs on different backends is the fastest empirical answer to "did the new model actually make this workflow worse." That comparison, run per workflow rather than per benchmark, catches regressions generic evals miss.
The evidence trail and the control stack are complements, not substitutes. One tells you what happened; the other decides whether it happens again.
Frequently Asked Questions
What is a JSONL agent trace? One JSON object per line, appended in execution order, recording each step of an agent run: tool calls with normalized arguments, results (hashed or summarized), reasoning markers, and timing. It is the cheapest complete record of what an agent actually did.
How do you diff two agent runs? Pick a known-good "golden" run closest to the failure, canonicalize both traces (sorted keys, rounded floats, redacted IDs), then walk them in step order until the first divergence. The first differing tool call or argument set is usually the one that went wrong.
What can replay catch that observability dashboards miss? Silent wrongness — an agent completing a task through a sequence of individually healthy calls that produces a worse outcome. No error fires, so no dashboard blinks. A diff against a golden run surfaces it in one step.
Is replay a replacement for guardrails? No. Guardrails are policy enforcement before execution; replay is forensics after execution. You need both: one to bound the blast radius of a bad call, the other to find it fast when a bad call gets through.
How much storage does JSONL tracing cost? Roughly a few hundred bytes per step with hashing; a busy agent doing 10,000 steps a day generates on the order of megabytes. Compression and retention policies make even long histories trivial.
You built it. We optimize it.