Closing the Loop From Observability Signal to Agent Improvement Without Guesswork

A production trace-driven harness engineering workflow turns live failures into layer-specific diagnoses, reviewable changes, and replay-tested fixes. Closing the loop from observability signal to agent improvement requires all three, supported by complete instrumentation. A trace shows what happened, but only attribution and validation establish what to change and whether that change addresses the failure.

Why observability data alone does not close the loop on agent failures

A production trace shows that retrieval gave the model an outdated policy document before it wrote its reply. The spans establish the sequence of events. They do not, by themselves, settle whether search returned old guidance or the model misused current guidance available in its context. That determination requires analysis of the evidence at the relevant steps. The distinction determines which layer needs a repair.

An agent can finish within its latency budget while returning the wrong answer, because latency and token count measure execution. In a longer workflow, a wrong retrieval can alter the plan, a bad tool argument can corrupt a result, and a sub-agent handoff can lose context. By the time the failure becomes visible, downstream steps may already have acted on bad data.

More trace visibility does not resolve that ambiguity. A trace viewer can support debugging of an individual run without providing a reliable way to measure quality across thousands of runs. Step-level and trace-level metrics provide that broader assessment, but they still need a process that connects a detected failure to its cause. Otherwise, teams accumulate evidence without a disciplined basis for choosing an intervention.

What agent-aware instrumentation must capture before attribution is possible

LLM-call tracing alone leaves agent execution incompletely described. Tool instrumentation must capture the tool name, version, exact arguments, raw output, and success or failure. State instrumentation must record changes and side effects, including database writes and API mutations. Reasoning-to-action traces must link the reasoning step to the tool selection, and recovery instrumentation must show retries, silent failures, and fallback paths. These records establish what the agent changed.

Recovery behavior needs its own execution history. A tool failure can cascade silently through a multi-step workflow and become visible in its execution history only after several later actions. A simple success/failure log cannot adequately distinguish graceful recovery from repeated attempts that never resolve the problem. Retry counts, fallback decisions, and loop iteration spans make those paths distinguishable. Without them, the visible error can be mistaken for the initiating fault.

A shared trace ID must run through every sub-agent hop so that attribution can follow a failure from one agent to another. Detailed local spans cannot reconstruct that relationship reliably if the connection between agents is missing. The captured context must also include both the core input and the prompt template wrapper. Attribution and evaluation need the system message, few-shot block, and tool schema that the model actually received.

Why locating the failure step is a distinct problem from detecting the failure

Detection establishes that a run violated an expectation. Attribution identifies the step and layer responsible, which may be well before the point where the violation surfaced. The responsible layer can be the prompt, tool, workflow, memory, model, or product logic. An outcome score does not distinguish among them. A useful diagnosis must connect its proposed cause to evidence in the trajectory.

AgentRx illustrates a rigorous approach to this problem. Its 2026 evaluation reports an average improvement of 75% in step localization over prior approaches. The framework synthesizes constraints and evaluates them step by step to pinpoint the critical failure step in a failed trajectory. Its auditable validation log records constraint violations, giving the diagnosis an inspectable basis. The important design property is the connection between the localized failure and the evidence supporting it.

AgentDebugX is another example from 2026 research, with strategies that weigh the cost of analysis against how much it can resolve. Inexpensive heuristics and single-pass whole-trace reading come first, followed by binary search, per-step inspection, and budgeted ensembles for harder cases. That escalation addresses a weakness of one-shot LLM attribution. A judge can settle on a plausible diagnosis early and leave critical evidence in a long trajectory unexamined. Longer task horizons make that shortcut more consequential.

The same framework specifies useful production design criteria. These include low-friction capture, a portable OpenTelemetry export path, and typed diagnoses containing root cause, evidence, confidence, and fix. Local-first storage and explicit scrubbing before sharing govern how trace data gets passed on for analysis. Cost-aware selection keeps deterministic triage free while making deeper LLM analysis opt-in. These research approaches illustrate why attribution needs more structure than a single judgment over a completed run.

The three failure categories that attribution must distinguish between

Workflow loops leave recognizable patterns in execution spans. Repetition of “same tool + same args + same error” points toward a tool outage or schema mismatch, while “bouncing between two tools” indicates a planner that cannot choose. A “ping-pong agent handoff” instead indicates unclear ownership. Each iteration has corresponding reasoning, action, and observation spans, so the pattern can be seen in both production traces and an evaluation pipeline. Teams can operationalize detection with evaluation thresholds, cohort-level alerts, and CI gates that block a prompt or tool-schema release when the same scenario regresses.

Prompt-template drift is quieter because curated inputs can remain stable while their wrapper changes. The model receives those inputs inside a system message, few-shot block, and tool schema. That wrapper is the prompt template. An evaluation that tests only the core inputs does not test the template and can miss the layer changing between deployments.

Tool schema drift has a statistical signature. The signal is the distance between an embedding centroid for the golden set and a rolling centroid over recent production traces. Separation indicates that the evaluation distribution no longer represents the inputs the agent is sending to the tool. Loops are quickly visible in span counts, while tool schema drift is visible in the changing distribution. Template drift remains invisible unless the wrapper is evaluated, so the three categories require separate detection logic.

Why fixing the harness is the highest-leverage intervention for most agent failures

The harness is the primary engineering lever around the model. Its prompts, tools, memory, skills, workflows, and orchestration logic determine what the agent can accomplish, and changing them is lower-cost and faster to validate than retraining. Harness engineering revises these components from execution feedback while leaving model weights unchanged. The resulting changes can be reviewed, audited, and rolled back. That makes a layer-specific diagnosis actionable as a bounded engineering change.

Google Research’s Harness², open-sourced in October 2026, takes this approach. It exposes eight components as editable files, namely the system prompt, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins. Google’s research reports gains across professional benchmarks from harness-only changes. Independent replication is still in progress, so those results should not be treated as a guarantee for a particular deployment. They nevertheless support the broader case for treating the harness as an optimization target.

The Recursive Harness Self-Improvement preprint reports a related result in 2026. After iterative harness improvement, low-reasoning-effort agents outperformed the maximum-effort configuration while spending up to 60% less on inference. The engineering explanation centers on prompt structure, tool interfaces, context compaction, and recovery logic, which can enable or prevent a long-running agent’s completion. More reasoning effort alone does not fix those parts around the model.

This work is tractable, but it is not permanent. The best harness is tied to a specific backbone, so its engineering cost returns with each model release. Teams need a repeatable process. Harness² also identifies additional API costs, security considerations, and human-review needs when changes are automated, especially when agents can modify their own guardrails. Reviewability is a condition of the repair strategy.

Why shipping a harness change without replay validation is still guesswork

A diagnosis makes a proposed change defensible. Agent failures depend on inference that is not bitwise reproducible, tools that read changing state, and trajectories that a fresh run rarely repeats. If the original conditions have changed, a successful run does not establish that the repair addressed the incident. Validation must preserve enough of the recorded execution to test the proposed intervention.

Chronicle, presented by Chawla & Koul at EMNLP REALM 2026 in October in Budapest, addresses this reproducibility problem. It records a run at its non-deterministic boundaries as immutable envelopes and replays from that record. Its central operation is cut-point replay. A chosen subset of boundaries is served from the record, while the complementary subset executes live with new code. This turns a recorded incident into a deterministic regression test that can run in CI without additional model calls.

Chronicle showed how cut-point tests worked on six incidents, where they killed 51 mutants of the guarded tools while full-stub replay killed none. In that evaluation, replaying everything as a stub failed to expose regressions that selective replay caught. The comparison distinguishes reproducing a recorded path from testing changed code against it.

Incident replay and golden-set replay serve different purposes. The former tests the failure that motivated the change, while the latter assesses related cases through the current production pipeline using expected outputs or rubric scores. Scheduled golden-set replay, run daily or on every deploy, gives an aggregate quality signal, and a score that keeps falling is an early drift signal. That set must include the actual template wrapper so that teams can see changes in guardrails, few-shot blocks, or tool schemas. Passing only the golden set does not demonstrate that the specific root cause was repaired.

How do engineers use production traces to improve an agent in practice

Engineers can use the following sequence as a diagnostic checklist for finding where their current loop breaks. The first checkpoint is complete instrumentation. The record must include tool identity and version, arguments and output, state changes, reasoning-to-action links, and recovery behavior. It must also preserve the prompt wrapper and a shared trace ID across sub-agent hops. If those records are absent, the gap precedes analysis because the evidence needed to attribute the failure was never captured.

The second checkpoint is failure detection at both levels. Step-level metrics assess tool selection accuracy, argument validity, and retrieval relevance, while trace-level metrics assess task completion, policy adherence, and context continuity. A process that records only the final outcome stops here. It can find runs that failed without finding where behavior first went wrong.

The third checkpoint is layer-specific root-cause attribution. The diagnosis must assign responsibility to the prompt, tool, workflow, memory, model, or product logic and retain the evidence supporting that assignment. The analysis must also work for long trajectories. If a one-shot LLM judgment is the entire attribution process, the missing capability is deeper examination when the initial account leaves evidence unresolved.

The fourth checkpoint is a targeted harness repair. The proposed change must be concrete, inspectable before application, and documented against the trace that motivated it. A black-box auto-fix prevents that review. Recurring signatures across many traces also call for structural repair rather than isolated patches, because a repeated schema or workflow failure is evidence of a design problem.

The fifth checkpoint is replay validation before shipping. The candidate must pass both the recorded incident test and the curated golden set before it becomes a candidate for production. If either test is missing, the gap is validation, regardless of how persuasive the diagnosis appears. The release decision must rest on evidence that the changed harness addresses the live failure without regressing the related cases.