Signal Fidelity Trade-Offs When Replay Eval Replaces Benchmark Suites for Prompt Changes

Before deploying an agent prompt change, replay actual historical conversations through the production harness and score both the execution path and the final response. Replay eval gives distribution-matched evidence that synthetic benchmarks cannot provide, but how much it can be trusted depends on serving stability, how carefully scores are set, and the quality of the traces. The evidence supports using replay for production-specific validation while retaining benchmark suites for reproducible regression coverage.

Why static benchmark suites lose fidelity as production agent traffic evolves

A static benchmark preserves the task distribution that existed when its cases were written. Production does not stay still for that snapshot. New user phrasings, updated tool schemas, and revised system prompts change what the agent encounters and how it must respond. As those changes accumulate, a passing score becomes weaker evidence that a prompt edit will work on current traffic. The suite can still detect regressions in known cases, but those cases represent a shrinking portion of what the agent encounters in production.

Contamination and saturation weaken the signal further. When benchmark questions appear in training corpora, a model may remember the answers, leaving its ability to do what the test asks unclear. When frontier models all score near the top, the suite can no longer show which differences between them matter. Either problem can go unseen in the reported number. Prompt edits introduce another mismatch because a changed template may define a contract the dataset and rubric were never written to assess. Previous scores then cease to be comparable without rerunning the suite under the changed contract.

How to test agent changes against real production traffic before deploying

Replay eval substitutes production traces for curated benchmark tasks. A full trace includes model calls, tool calls and their arguments, retrieval context, agent state, and the final response. That record changes the unit of evaluation. Rather than grading only a prompt-answer pair, replay can assess the sequence of decisions that produced the answer. Its input set reflects traffic that actually arrived, although its relevance still depends on which traces were retained and when they were captured.

OpenAI’s agent evaluation tooling organizes evaluation around traces, graders, datasets, and eval runs. This structure connects evaluation to production debugging. Final-answer scoring asks whether the result was correct, trajectory scoring examines the full step sequence, and per-turn scoring examines individual interactions. Replay makes trajectory scoring possible before release. Replay can identify which step failed, under which tool call, and with which prompt version. Across repeated evaluations, the same approach can also investigate prompt drift, model drift, and eval-score drift rather than limiting assessment to a frozen validation event.

The harness effect that makes distribution matching matter most for prompt changes

CORE-Bench results show that the same model can score very differently depending solely on the agent harness. The harness’s effect can exceed the differences that prompt-change evaluations are designed to detect, so a model’s published benchmark score says little about how a prompt edit will behave inside a particular production system. The relevant comparison holds the application context close to the one in which the change will run.

Replay makes that connection by testing the actual harness against production traces. This is especially important for prompt changes because an edit alters how the model parses context, selects tools, and structures intermediate reasoning. Those changes appear in the trajectory. A correct response can hide a path that costs too much, is unsafe, or is wrong in some other way. Distribution matching therefore means more than choosing realistic questions. It also means preserving the execution context in which a prompt influences the agent’s decisions, so that a score reflects the application.

Where replay eval’s own signal breaks down — rerun noise versus prompt-perturbation noise

Real inputs do not make an evaluation quiet. The Noise Floor Audit found that reruns at frozen decoding were nearly deterministic, with ever-flip fractions no higher than 2.7% and mean run correlations above 0.96. Yet semantics-preserving prompt perturbations produced a noise floor many times larger than rerun paired standard deviations. The distinction is consequential for replay-based prompt validation. Repeating the same run estimates rerun variability, but leaves open how much the result depends on the wording change being tested.

More replicates alone therefore cannot show whether a small change in score comes from the intended improvement or from perturbation sensitivity. A separate complication arises when a model swap causes trajectories to diverge at the first post-fork action, even without a prompt change. Branching unrelated to the edit can then confound the comparison. The audit also found that failure character shifts across endpoints, from malformed outputs to well-formed-but-wrong calls. Aggregate accuracy hides that distinction. A stable total can therefore obscure a change in what fails, while a changed total may not establish why it changed.

How quantization and serving configuration make replay’s counterfactual unstable

Temperature-zero decoding does not guarantee a stable counterfactual. In the reported FP8-quantized controls, trajectories diverged on 90–96% of forks at edit distances of 0.49–0.67 under identical decoding settings. AWQ-served controls, by contrast, stayed almost the same. These results identify serving configuration as a separate source of variance from prompt perturbation. A replay score can move because the infrastructure changes the trajectory, even when the comparison appears to hold decoding constant.

This instability affects what a prompt comparison can establish. If an unchanged control branches substantially, movement in the edited run cannot confidently be assigned to the edit alone. Replay against live or near-live infrastructure exposes this dependency in a way that a fixed benchmark dataset does not resolve. Fixed inputs can still lead to execution that is not deterministic. The engineering requirement is to audit the quantization scheme, whether batching is deterministic, and whether replay follows the same inference path as production. Those checks address the stability of the counterfactual before the score is treated as evidence about the prompt.

Coverage gaps replay eval cannot fill — tail inputs, schema drift, and tool failures against live systems

Replay covers observed traffic. A prompt change can introduce behavior that historical traces do not contain, while an external API can change after those traces were captured. Zhu et al.’s study found that API Misuse accounts for 32.97% of bug reports in modern LLM-agent frameworks. An added required field or deprecated endpoint can make a previously valid tool call fail. CI-side schema snapshot testing addresses that gap by recording tool schemas, diffing them on each run, and failing on breaking changes.

Jailbreak attempts, policy violations, and user frustration signals need per-turn production coverage at scale, while offline LLM-as-judge scoring on sampled traces can only come close to that coverage. Within replay itself, response-only scoring leaves tool calls and retrieval failures unexamined. Capturing a trajectory is therefore insufficient if the grader ignores it. In multi-step agents, a single error can propagate through subsequent decisions until the visible failure sits far from its origin. Replay may find the failed outcome without finding the component that caused it.

Why root-cause attribution inside replay scoring is harder than it looks

That distance makes attribution a search problem. A grader must examine evidence across the prompt, tools, workflow, memory, and model, test competing explanations, and reject those contradicted by the trace. A plausible explanation is not enough by itself. As execution logs grow and spread across more places, LLM-based graders can settle on what seems to be the cause before looking through all the evidence.

A tool argument mismatch illustrates the consequence. If the grader labels the resulting trajectory failure as a prompt defect, the next prompt edit responds to the wrong diagnosis. The feedback loop can then become misleading even though the evaluation correctly detected that something failed. Replay systems need typed diagnoses that distinguish root cause, evidence, confidence, and proposed fix. These fields make the attribution claim explicit rather than burying it inside an aggregate rubric score. They expose what evidence supports the proposed explanation and what uncertainty remains.

The data-governance cost of replay eval that benchmarks do not share

Production traces can hold secrets, personal data, and the contents of system prompts. Curated benchmark datasets are scrubbed before distribution, whereas production records arrive before that work has been done. Replay therefore creates an operational obligation at capture. A corpus containing PII or API credentials presents legal and contractual risk. Treating trace ingestion as ordinary logging leaves that obligation unresolved.

The practical requirement is local-first trace storage with explicit scrubbing and a clear boundary between retained and discarded data. Those decisions need to be made when traces are captured, before scoring. They also affect which evaluations remain possible because trajectory scoring depends on the context surrounding each action. Governance and signal fidelity therefore need to be handled together at the same stages. Excessive scrubbing can remove evidence needed to reconstruct the failure, while insufficient scrubbing can leave the corpus non-compliant. Resolving the resulting tension requires keeping the execution evidence needed to reconstruct the failure while discarding data that should not be kept.

Trace quality as a precondition — what makes a production corpus reliable for replay

To address these dependencies, engineers should treat trace quality as an engineered property. Capture completeness is the baseline for trajectory scoring. The record must include intermediate model calls, tool calls, retrieval context, agent state, and the final response. Scrubbing must preserve the evidence that those evaluations require. OpenTelemetry GenAI conventions provide a portable, vendor-neutral record for export, scoring, and comparison with past releases, but replaying application state requires tooling beyond the conventions themselves.

Rubrics need the same versioning discipline as prompts. Teams should treat prompt changes like schema migrations, retaining the rubric version with the traces it was designed to grade. If the rubric changes, historical traces need rescoring before their results become comparable. Changes to system messages, few-shot blocks, and tool schemas also require rubric revalidation because the grading contract may have shifted. This discipline keeps changes to the measuring instrument from becoming another hidden reason for a score to move.

Cost-aware analysis makes that discipline practical at trace scale. Deterministic schema checks, loop detection, and argument validation should run on every trace, while deeper LLM scoring should be opt-in and targeted at flagged traces. This division also supports attribution because a concrete argument-validation failure supplies evidence for a typed diagnosis. It spares each trace a costly grader investigation with no set endpoint. The deeper scoring layer can then focus on cases where deterministic checks indicate that further explanation is needed.

Running replay eval and benchmark suites as complementary layers, not substitutes

Replay and benchmarks answer different release questions. Replay owns distribution-matched trajectory validation in CI and pre-release evaluation, while benchmark suites provide reproducible regression coverage and cross-version comparison. A fixed, scrubbed suite gives a stable reference that a changing production corpus cannot give on its own. Conversely, that reference does not establish how the current harness behaves on current traffic. Neither result can stand in for the other.

The operational pattern is signal-gated promotion. A team evaluates the change on production traces, validates it against historical traces, and compares behavior on live traffic before promotion. Benchmark coverage still checks known cases, while per-turn production evaluation checks for drift, new failures, and the long tail that no dataset contains. This arrangement links a broken trace to a proposed fix while leaving room for deployment risks that offline replay may miss.

Benchmark suites establish whether a prompt change regressed known cases, and replay tests whether it improves behavior on the observed production distribution. When the results diverge, that disagreement signals that the benchmark suite may no longer represent production, making the gap between the two evaluations part of the release assessment itself.