Curating a Production Trace Corpus That Makes Replay Eval Meaningful

Replay-based eval tests agent changes against real production traffic before deployment by running candidate changes against curated historical trajectories. Meaningful coverage requires failure diversity, complete layer coverage, normalization, root cause attribution, and harness versioning. This makes production conversations useful for validating prompt edits and regression testing agent behavior beyond what synthetic benchmarks or final-answer comparisons can reveal.

Why raw trace volume is not the same as replay coverage

Collecting traces records activity, but it does not establish replay coverage. Production traffic is skewed toward happy paths, so an uncurated collection over-represents behavior that already works. Rare intents and edge cases make up less of the archive even though they reveal the failures a regression suite needs to catch. More traffic can deepen this imbalance without filling the gaps.

Those gaps are difficult to reconstruct after an incident. LLM inference is not bitwise reproducible even at temperature zero, tools read external state that has since changed, and a rerun rarely repeats the same multi-step trajectory. The recorded trace is therefore the durable account of what happened. If that record is incomplete, a later run cannot reliably supply the missing evidence.

Answer-level evals impose a second limit on coverage. They inspect the final output but miss cascading drift in tool-call sequences and intermediate reasoning. Full-trace replay exposes the chain reaction, allowing regression tests built from actual user journeys to catch tool-call sequencing breaks, latency regressions, and multi-turn context drift that golden-answer suites miss.

What the corpus must structurally contain before replay becomes trustworthy

A replay corpus is a structured collection of trajectories with enough context to score behavior and interpret changes. Each row must include the user goal, system prompt version, expected tool sequence when one exists, realized tool calls with arguments and returns, intermediate reasoning, final response, and outcome label. The trajectory is the unit of evaluation. A final answer detached from its tool calls is not an equivalent test case.

This structure follows from what an eval does. An eval maps a trajectory to a score, so its interpretation depends on the components being present. Without intermediate tool calls, the record cannot show how the agent reached the result, and without the system prompt version, it is unclear what the agent was meant to do. Either omission makes the regression anchor brittle because a score cannot explain evidence that the corpus never retained.

Complete rows still need deliberate stratification by tool, argument-edge-case bucket, and error code. The lesson for corpus design is to favor hard inputs, edge cases, and rare intents while retaining happy-path coverage for sanity checks. Most of the useful eval signal comes from those harder inputs. Failing production traces should enter the promotion process on a regular cadence rather than wait for an occasional manual curation pass. That process needs a further admission requirement, because a complete trajectory can still contain runtime values that make it unsuitable for reproducible replay.

Normalization as the unglamorous prerequisite for deterministic replay

Normalization removes incidental variation without erasing the behavior under test. Timestamps, session IDs, and runtime-specific values that change across runs without reflecting a meaningful behavioral difference must be scrubbed before a trace enters the corpus. The full message sequence, tool-call arguments and their exact values, runtime branching decisions, and system prompt version must all stay as they were. The distinction is semantic.

A value belongs on one side of that distinction according to its role in the test. Removing an argument that shaped a tool call would erase part of what happened, while keeping a session value that has no bearing on the behavior would tie the test to the original session. Normalization calls for engineers to judge what replay is meant to keep the same.

Chronicle's benchmark reports full replay that is bit-stable across 20 repetitions. That property should not be confused with requiring candidate agent behavior to be bitwise identical to the record. Fidelity thresholds are the right CI primitive for checking whether behavior stays within an acceptable range.

Without normalization, flaky results invite manual overrides and discourage investigation. Multi-agent systems compound the risk because silent drift in one agent can propagate through the pipeline. Answer-level normalization cannot expose that propagation, so normalization must operate at span level across the full trace tree.

The three failure categories a corpus must cover to be honest about production

Tool schema drift, prompt ambiguity, and workflow loops form a practical audit checklist. A corpus with no examples of these categories leaves replay blind to major production failure modes. Each category tests a different weakness in the agent's execution.

Tool schema drift can produce a malformed-but-plausible response without raising an exception. That response passes downstream until wrong outputs become visible to a person. Earlier signals include unknown payload fields, missing required fields, fallback-to-default-value events, and mismatches between the schema version expected by the planner and the version served at runtime. Traces must retain those signals. Otherwise, replay can accept the wrong contract and allow a schema regression to pass silently.

When a prompt lacks a verifiable done-condition, the agent has no internal signal that the goal has been met and can continue after the task is complete by outside measures. Labeled traces of those runs provide coverage for prompt changes that reopen the same gap. A final answer that looks right cannot by itself show whether the agent stopped when it should have.

Workflow loops appear as repeated tool calls, unsuccessful repairs of self-introduced errors, or endless edits. In multi-agent systems, two agents can loop because neither has a termination condition satisfied by the other's output. Full-agent-graph traces are necessary to capture that interaction. Records of loop onset and termination, however forced, supply the ground truth for testing whether a change prevents recurrence.

Why root cause attribution must be resolved before a trace earns its place in the corpus

A failed outcome identifies an incident. Without attribution, replay may detect a behavioral change but cannot direct the team toward the component that needs intervention. That makes the failing test unactionable.

Attribution is difficult because the visible failure often occurs after the action that caused it. Relevant information can be sparse, spread across distant actions, and disconnected from the final error. A complete trajectory provides the evidence for investigation, but completeness does not itself resolve causality. The admission gate must distinguish the symptom from the fault that produced it.

The responsible component may be the model, the harness, the environment, or the grader. Within the harness, attribution can point to the prompt, tool, workflow, or memory. These distinctions determine the intervention. Harness failures show where the harness needs to be redesigned, while environment issues call for repairs to the tool contract. Misclassifying either as a model problem directs effort away from the layer the team can usefully change.

Existing observability tools can replay execution traces but do little to help find the root cause or decide how to recover. Chronicle, by Tisha Chawla and Susheem Koul at REALM 2026, connects recorded incidents to candidate fixes through cut-point replay. Its usefulness depends on the prior diagnosis, because the team must know which layer the incident belongs to before choosing where new code should execute.

How to test agent changes against production traces with Chronicle's cut-point replay

Chronicle distinguishes reproducing a run from testing a fix. Conventional record-and-replay reproduces the recorded execution, but a regression test must expose a candidate change to the conditions of a past incident. Cut-point replay makes that distinction operational. It turns the trace into a regression anchor rather than an execution that can only be watched again.

The system records an agent run at its non-deterministic boundaries as immutable envelopes. During cut-point replay, a chosen subset of boundaries is served from the record, while the complementary subset executes live with new code. This arrangement validates the changed component against the recorded conditions of the failure. Attribution determines which boundaries should run live, so the test does not substitute an arbitrary experiment for the diagnosed incident.

A per-name call-count check on stubbed boundaries guards an important part of the execution contract. A candidate fix cannot pass unnoticed by invoking a stubbed tool fewer or more times than the recorded incident required. Call structure is part of the test. This check complements outcome evaluation by detecting a change that a plausible final response could conceal.

Recorded envelopes also replace hand-written mocks with evidence drawn from the original run. Their reliability depends on upstream normalization, since un-scrubbed session values can make them unsuitable for CI replay. Complete records establish the conditions, attribution selects the intervention, and cut-point replay tests that intervention without rerunning every non-deterministic boundary live.

Layer-level versioning as the discipline that keeps the corpus honest over time

A trace must identify the harness state that produced it. Without that context, a replay regression cannot be distinguished from the expected effect of a deliberate prompt or tool update. The problem grows as the harness changes. A corpus can preserve each incident faithfully while becoming misaligned with the system its tests are supposed to represent.

Prompts, tools, routing logic, memory configuration, and skill definitions should therefore be versioned, traceable, and rollback-capable. This extends the discipline used for application code to the components that govern agent behavior. Version metadata gives each corpus entry a defined relationship to those components.

Promoting failing production traces addresses the manual curation bottleneck that leaves conventional eval suites stale, but it can also mix traces from different harness versions as though they describe the same system. Recency does not prevent contamination. A newly captured incident still needs version context before its comparison with older behavior is interpretable.

The CI gate should assert independent thresholds for tool selection, argument extraction, result utilization, error recovery, plan coherence, and task completion. An aggregate score can hide a regression in one dimension behind headroom in another.

Closing the loop: how live traffic replenishes the corpus without reintroducing curation debt

Production observation closes the loop when it applies the same rubrics to live traces as span-attached scores. Failing traces can then auto-cluster into named issues and enter the corpus as candidates. Promotion is not automatic acceptance into the permanent regression set. Normalization and attribution remain admission requirements, while version metadata preserves the harness context needed for later comparison.

For multi-turn agents, per-turn scoring must give way to conversation-level and outcome metrics. A series of turns that each pass can still lead to a failed journey, so scores for each turn on its own can give false confidence. The rubric attached to live spans must match the rubric used to score the offline corpus.

Regularly refreshed production datasets address a limitation of static benchmarks. Real users find failure modes that test authors did not foresee, providing a source of new coverage. Edge cases and failure traces belong in the permanent regression set, while happy-path traffic informs coverage monitoring without displacing failure diversity.

The resulting corpus must remain representative of current production behavior, stratified by failure type, normalized for deterministic replay, attributed by responsible layer, and versioned by harness state. All five properties are required as the system evolves. Maintaining them is the ongoing engineering work that lets historical conversations test tomorrow's prompt, tool, and workflow changes before they ship.