Diagnostic Instrumentation for Prompt Versus Tool Call Failure Isolation in Production Agents

AI agent failure attribution by harness layer needs span-level tracing that keeps a record of the inputs, outputs, and evaluation signals for each layer. Distinguishing a bad prompt from a broken tool call is fundamentally an instrumentation problem. The diagnostic pattern is to capture causal spans, evaluate each layer separately, validate tool boundaries, and replay ambiguous failures before changing the harness.

Why prompt-versus-tool attribution fails without span-level instrumentation

A tool call reports success, the trace looks healthy, and the agent proceeds with malformed data. Nothing throws an exception or triggers an alert. The model treats the response as valid, and each downstream step builds on corrupted state even though the step makes sense on its own. Standard logging records that the request succeeded and how the run ended, but leaves out where the failure began.

Reviewing the trace as a whole cannot reliably show that difference. Tool responses account for a large majority of tokens in an agent trace, while system prompts account for a small fraction. Prompt-level signals are therefore swamped by tool-layer noise, even when a prompt caused the failure. An ambiguous tool description, for example, can lead the agent to select the wrong tool or supply incorrect parameters. The error is visible at the tool boundary.

Deepchecks’ agent evaluation research identifies the underlying measurement gap. Distinguishing an instruction defect from a malformed response requires records of what each layer received and produced.

The four failure categories where prompt and tool causes are hardest to tell apart

Tool schema drift creates a mismatch between the interface the agent expects and the interface the server serves. MCP tool schemas can drift without description changes or version bumps, so the description alone does not reveal the change. The agent may seem to misunderstand a tool it otherwise knows how to use. Yet the defect may be in the interface, making the visible tool error insufficient evidence of its origin.

Prompt and workflow drift produces a different ambiguity. A prompt update in staging can cause production agents to degrade gradually. The agent still works in the narrow sense that it completes steps and returns answers. Without version-tagged evidence, attributing that slow deterioration to either the prompt or changing tool behavior is difficult.

Workflow loops and step repetition blur the boundary further. Deepchecks’ research and the MAST taxonomy identify step repetition as a leading failure category, accounting for 15.7% of all failures in multi-agent systems in 2025. An agent retries the same tool call with small changes to its parameters that look like progress. The repetition can reflect a reasoning failure or an attempt to recover from an unhelpful tool response. The repeated call alone does not distinguish them.

Hallucinated tool calls from ambiguous descriptions reverse the apparent direction of blame. An underspecified description leads the agent to choose the wrong tool or construct incorrect arguments, producing a tool-layer error from a prompt-layer cause. Deepchecks’ diagnosis work identifies localization between the LLM span and Tool span as a core attribution challenge. The symptom and cause occupy different layers. Across these categories, locating the visible error is not the same as locating the responsible decision or response.

What a span hierarchy must capture to make attribution possible

Each agent run needs a root span with causally linked child spans for model reasoning steps, tool invocations, memory reads, and sub-agent activity. These operations cannot be collapsed into a single parent record. A failure introduced by one sub-agent may propagate through later steps, and the hierarchy preserves the relationship between that origin and its effects. OpenTelemetry provides a standard for tracing, but the diagnostic value comes from what the spans capture.

Every span needs the exact inputs its layer saw at invocation and the raw outputs it returned. For a reasoning step, the input includes the active prompt, with its version preserved for later evaluation. For a tool call, it includes the actual arguments. Summaries are not enough for this purpose. A summary can omit the malformed field or ambiguous instruction needed to distinguish a bad decision from a bad response.

Latency and token counts also belong to individual spans. Per-span latency locates a slow tool response, while per-span token counts help identify context saturation at a particular step. Tool-call spans must also record the schema the agent believed was active at invocation. That historical record supports retrospective drift detection even when the current schema differs from the one used during the failure.

Each span must identify whether it represents prompt or reasoning activity, a tool call, a memory operation, or a sub-agent handoff. Without those tags, records with inputs and outputs do not reliably identify their role. The tags make filtering and evaluator placement mechanical. They also preserve the distinction between where a failure was detected and which layer produced the evidence under examination.

How do span evaluators pinpoint which agent layer caused a failure?

Run-level evaluation establishes whether the task succeeded, but span-level evaluation tests the operations that produced that result. The two answer different diagnostic questions about the run. A tool-call evaluator asks whether the returned data was valid for that call. A reasoning-step evaluator asks whether the step followed its instructions and produced an aligned next action.

At the tool layer, schema conformance must be checked on every invocation, including apparently successful calls. Output validity then tests whether the payload is semantically coherent for the expected result. An empty array or null field may fit a schema but give the model nothing it can use. Structural conformance cannot establish semantic validity on its own. A latency anomaly signal adds a separate test, flagging calls that exceed the tool’s p95 and may have returned timeout-driven partial responses.

At the reasoning layer, evaluation compares the recorded reasoning with the tool call made in the next child span. Reasoning-action mismatches fall under inter-agent misalignment, which accounts for 13.2% of multi-agent system failures in the notes’ taxonomy. Instruction adherence checks whether the output follows the system prompt that was active for that span, so the record must include that version of the prompt. Loop detection compares the current tool-call decision with the one two spans earlier.

If the tool evaluator fails and the preceding reasoning evaluator passes, the evidence points to the tool layer. If reasoning fails while the tool evaluation passes, it points to the prompt layer. When a tool failure is followed by a downstream reasoning failure, the tool remains the likely origin. Two failing evaluators alone do not establish that direction, so unresolved cases require replay-based analysis.

Schema validation hooks as a dedicated instrumentation layer for tool-layer isolation

Schema validation belongs at the tool boundary, before the data that comes back reaches the model’s next reasoning step. A post-hoc evaluator can diagnose contamination but cannot prevent it. PreToolUse hooks validate the arguments the agent is about to send, while PostToolUse hooks validate the response before it enters context. Both operate as lifecycle events outside the model’s control. The agent cannot skip or override them.

Schema fingerprinting makes interface drift visible at that boundary. The hook computes a fingerprint of the schema currently served by the tool server and compares it with the schema recorded at the last successful invocation. It also preserves the schema seen for the current call. A mismatch raises a drift signal before the response reaches the model, rather than waiting for downstream reasoning to expose the change. An unchanged description is not evidence of an unchanged schema.

The hook writes the schema version, fingerprint hash, validation result, and specific failing field into the span. The result distinguishes pass, fail, and warn states. These records make the span self-contained for retrospective diagnosis, even after the server’s schema changes again. Without the recorded version and fingerprint, an engineer examining the current interface may be unable to reconstruct what the agent encountered.

This layer has a precise limit, however. Schema validation checks how the data is structured, while output validity evaluation checks what it means. A malformed response or truncated JSON belongs to structural validation, but a structurally valid response can still be semantically wrong. An empty payload is a schema violation only when the schema rules it out. Keeping both signals distinct prevents a passing hook from being mistaken for proof that the tool supplied usable information.

Spectrum-based failure attribution across repeated trace replays

When prompt and tool evaluators both flag across different replays, a single trajectory may not settle attribution. Spectrum-based analysis compares repeated executions of the failed task, examining which actions recur in failing trajectories relative to passing ones. An action receives higher suspiciousness when its presence is more consistently associated with failure. This produces a ranking.

Famas, from Nanjing University and The Hong Kong Polytechnic University, introduced the first spectrum-based failure attribution approach for multi-agent systems in September 2025. It addresses a setting in which LLM-based attribution methods achieved action-level accuracy of less than 10%. Its core premise is that a failure-responsible action and its resulting state recur across repeated executions of the failed task. Systematic trajectory replay and abstraction make those recurrences available for comparison.

The method brings together patterns in agent behavior and patterns in action behavior. The first group records which agents are activated, while the second concerns the specific actions activated within those agents. That distinction narrows attribution beyond identifying an implicated agent. An agent that otherwise works well may have one action with a strong link to failing trajectories. Scoring both levels avoids treating all of that agent’s activity as equally suspicious.

For prompt-versus-tool isolation, the highest-ranked action provides evidence about the responsible layer. A tool-call action at the top of the ranking points toward the tool layer, while an earlier reasoning action points toward the prompt layer. The ranking supports only tentative causal attribution. Famas is a research result that establishes a rigorous conceptual model for replay-based diagnosis. Safe replay also requires wrapping tool side effects and substituting recorded responses for live calls so that destructive actions are not repeated.

Turning attributed failures into validated harness fixes without shipping blind

Once attribution identifies the responsible layer, the corrective change should target that layer. A prompt-layer defect calls for a harness change that revises the system prompt, tool description, or instruction that produced the ambiguous reasoning. A tool-layer defect instead calls for correcting the schema, updating the PostToolUse validation hook, or adding a fallback for the recorded malformed response pattern.

Workflow attribution requires a different intervention from either of those. A loop or drift pattern may call for restructuring the step sequence or adding a termination condition. The fix then belongs to workflow design, rather than to the prompt or tool individually. This distinction prevents the visible symptom from dictating the wrong change.

Before it ships, the proposed fix must pass deterministic replay using the past traces that showed the failure. Otherwise, the original diagnosis has not produced a validated correction. Span-level evaluators and schema validation hooks also belong in CI as regression gates. A harness change that degrades a previously passing span’s evaluator score should fail the build.

A recurring pattern calls for a structural harness fix rather than a one-off correction to the failing trace. The appropriate response may be a new hook, a revised tool description, or an updated evaluation case. Its scope is the class of failure revealed by attribution.