Root Cause Attribution Across Agent Harness Layers

The log says the task failed and the final answer's wrong. It doesn't say why, and that gap is the whole problem. Root cause attribution across agent harness layers is a search problem, not a classification problem: the evidence is sparse, buried early in the trace, and scattered across parts of the log that don't look related on the surface. Most teams treat it as a guessing game anyway, and that's the mistake this piece is about.

What actually happens is this. A retrieval agent pulls the wrong file at step 2. Steps 3 through 10 look fine, each one reasoning plausibly off the last, until step 11 produces an answer that's simply wrong. Nothing in the final output points back to step 2. The team staring at the trace has to decide: prompt, tool call, stale memory, a workflow branch that fired when it shouldn't have. Without a method, someone picks the layer that feels most likely, patches it, and ships. Sometimes that guess lands. Often it doesn't, and the team burns a sprint rewriting a prompt that was never broken, while the tool schema mismatch that actually caused the failure sits untouched, waiting to cause the same failure next week.

Misattribution doesn't just waste engineering time. It actively breaks things that worked. An engineer convinced the model reasoned badly will rewrite a prompt that was fine, bolting on constraints that degrade performance on cases the agent used to handle correctly. What matters isn't a failure flag. What matters is which layer caused the failure, and what evidence in the trace proves it.

Attribution across long traces as a fundamentally a search problem

Three properties make agent traces hard to read, and they compound each other. Evidence is sparse: the one or two spans that explain a failure sit among thousands that look completely normal. The root cause tends to originate early, in a step whose output looked locally fine when it ran. And the signals that matter are scattered, so the clue and the failure rarely sit anywhere near each other in the log.

Scale makes it worse. Multi-step agentic workflows commonly run to millions of tokens, and nobody is reading a million-token log line by line, no matter how much coffee is on hand.

Throwing an LLM judge at the trace and asking it to name the cause sounds like a fix. It sounds like a fix but isn't one. Scale AI's Continual Search paper (Raj et al.) found that standard one-shot judges settle on a plausible-sounding diagnosis early and stop reading, leaving the rest of the trace unexamined. Their fix was iterative search rather than a single pass, and it raised GPT-5.5's F1 score on MegaRCA-Mix from 0.349 to 0.498, a jump of more than 40%. That gain didn't come from a better prompt to the judge. It came from refusing to let the judge stop at its first guess, which says the one-shot approach is structurally wrong for this task, not just under-tuned.

Attribution has to work the way search works: propose a candidate explanation, look for evidence that would contradict it, and settle only once the alternatives have been ruled out, not merely outcompeted by whatever came to mind first. For an engineer, that means narrowing across candidate layers step by step, checking span-level evidence at each stage, and treating "plausible" as a hypothesis rather than a conclusion. AgenTracer found that state-of-the-art reasoning models score below 10% accuracy on root cause attribution without task-specific framing. Naive prompting of a judge model isn't close to solving this on its own.

If attribution is search, the search space needs a map. That map is a taxonomy of harness layers, each with its own failure signature.

The harness layers that define the search space

The harness is everything wrapped around the model that turns a language model into a working agent: the loop that drives it, the tool-calling interface, context and memory management, guardrails, tracing infrastructure. It's the plumbing, not the model weights, and that distinction matters more than it looks.

Adnan Masood's analysis of enterprise AI project failures found that 65% trace back to defects at the harness level. When an agent performs badly, the model is usually not the problem. The scaffolding around it is, and most teams look in exactly the wrong place first.

Six layers make up the attributable search space. The prompt layer sets standing behavior, task policy, and reasoning procedure in plain language; its failures are specification problems. The tool layer exposes external services through action schemas and argument formats; its failures are invocation and schema mismatches. The workflow layer runs the loop itself, gating phases, handling retries, passing state between sub-agents; its failures are coordination and sequencing problems. The memory layer carries state across time so the agent doesn't depend on what's still sitting in its context window; its failures are state management problems. The model layer is the LLM's own reasoning once everything around it checks out clean, and it's the layer to blame last, only after the rest are cleared. The product logic layer covers task definition, role assignment, and what counts as success in the first place; its failures are ambiguity baked into the spec.

These layers don't overlap in what fixes them. Rewriting a prompt does nothing for a tool schema that's drifted out of sync with the live API. Sending a team down the wrong layer isn't just wasted motion. It leaves the actual defect live in production while everyone congratulates themselves on a fix that never touched it.

Prompt layer failures as they appear in a trace

The prompt layer governs standing behavioral rules, reasoning steps, task framing, role definition, constraints, and expectations about output format. This is the single biggest category of failure, by a wide margin. The MAST taxonomy, validated across more than 1,600 traces, puts specification problems at 41.77%, covering role ambiguity, unclear task definitions, and missing constraints.

Prompt failures appear in a few recognizable shapes. An agent misjudges its own scope and acts outside a boundary the prompt never actually drew. It chases a plausible but wrong strategy because a stopping condition or output format was implied rather than stated. Or it skips a step the prompt's author assumed was obvious enough to leave unsaid.

The trace evidence has a particular texture. The model's reasoning spans read as internally consistent: this isn't confusion, it's confident pursuit of the wrong objective. The same failure pattern appears across different tool calls and different memory states, which points to a shared, upstream cause rather than something tied to one input. And the failure tends to appear early, in the agent's initial plan, before any tool call or memory read has even happened.

If the failure only appears with certain inputs, or changes shape depending on which tools got called, it's probably not a prompt problem. Once the prompt layer is implicated, the fix is a revision, a constraint added, a role clarified, and that kind of change can get tested against historical traces before it ever touches production.

Tool layer failures and the seven recurring error types

The tool layer is the contract between the agent and the outside world: action schemas, invocation formats, argument types, validation rules. ToolScan, a taxonomy from Kokane et al. (2024) presented at the Building Trust Workshop at ICLR 2025, names seven error types that keep recurring across tool-using agents.

An agent skips a call it needed to make. It sends the right schema with the wrong value. It gets the field name wrong. It sends the wrong type for an argument. It calls the same tool over and over with no change in state, a redundant loop going nowhere. It invents a function name that doesn't exist in the schema at all. Or it calls a real tool correctly and then fails to parse the output it gets back. A separate taxonomy built around REST APIs adds a few more angles on the same problem: wrong tool chosen outright, missing or hallucinated parameters, execution failures, misreading what came back.

Schema drift causes a lot of this: the tool schema baked into the prompt no longer matches the live API. The agent calls a tool that looks valid, with an argument that looks valid, and the API rejects it anyway because something changed on the other end. The error lands in the tool span, in the call itself, not in the reasoning that led to it.

Tool spans are the most legible evidence anywhere in the trace. They record the tool name, the arguments sent, the raw output, how long the call took, how many retries, whether it errored. A string of identical calls with no change in state between them is a retry loop the workflow never broke. An argument type mismatch or an unrecognized field name appears as an API error code sitting right there in the raw output, even while the reasoning span above it looks perfectly sound.

Check the tool span's raw output as a matter of routine. A 400 error the workflow treated as a soft failure and moved past anyway is a tool layer problem, full stop, not a workflow problem. If the arguments were right, the API responded correctly, and the agent still acted on the result wrongly, the failure has moved downstream, into reasoning or workflow.

How coordination breaks down in workflow layer failures

The workflow layer runs the loop, handling phase-gating, retry logic, the payloads handed between sub-agents, and the order in which tool calls and reasoning steps happen. MAST puts coordination failures at 36.94%, the second-largest category after specification problems, covering communication breakdowns, state that falls out of sync between agents, and objectives that conflict across parts of the system. Specification and coordination failures together account for 79% of production breakdowns in multi-agent systems. That leaves surprisingly little room for anything else to be the culprit.

The shapes here read differently from prompt or tool failures. Plan drift: the agent starts with a sound plan, but branching logic pulls it onto a path that no longer matches the original intent. Lost state synchronization: one agent hands a payload to another, but the handoff drops a piece of context the receiving agent needed, so it starts from an incomplete picture. Loops that never close: a retry condition that's never actually satisfied, so the workflow spins without progress. Phase-gate violations: execution starts before planning finished, compounding whatever error was already sitting in that unfinished plan.

State transition spans are where this evidence lives, showing state before and after each step, so a handoff that drops a field appears in the delta between the two. Plan drift becomes visible by comparing reasoning spans across time: the agent's stated plan at step 12 contradicts what it said at step 3, with no new information in between to explain the change. Escalating retries with no state movement are the signature of a loop that never closes.

Multi-agent systems raise the ceiling on how badly this can go. A UC Berkeley study cited in the AgenTracer research found failure rates in popular multi-agent frameworks reaching as high as 86.7%, with improper task decomposition and agents ignoring their assigned roles among the cited causes. If the plan stayed consistent and every tool call succeeded, but the final answer is still wrong, the workflow probably isn't where the problem lives. Look at memory or the model next.

Memory layer failures and the state continuity problem

Memory covers what the agent carries across steps: working memory, context edits, what it "knows" at any given point in a long task, what gets summarized away or evicted as context grows.

Memory failures get misattributed constantly. By the time a stale or dropped memory item causes visible damage, the agent's reasoning at that point looks like the problem, and the natural instinct is to blame the reasoning step rather than the memory read that fed it bad information several steps earlier.

Three patterns recur. Eviction errors: a summarization or compaction step drops a constraint or an earlier decision, and the agent's later reasoning stays internally consistent, just built on an incomplete picture. Stale reads: the agent pulls a value from memory that was accurate at step 4 but has since changed because of a tool call, and either the write never happened or the read skipped past the update. Context overflow: as an agent nears its context window limit without a managed way to compact it, earlier turns start dropping silently. Anthropic's server-side context editing tackles exactly this, cutting token consumption by 84% in a 100-turn evaluation, but agents without a mechanism like it lose context with no warning at all.

The evidence trail runs through state transition spans again: the before-and-after record at each step, including context edits, where a missing write or an outdated read appears in the gap between adjacent spans. A reasoning span that cites a fact contradicting the most recent tool output signals a stale read. A failure that only appears on longer runs, never on short ones, is close to a signature for context eviction. It occurs at the same logical step each time.

The diagnostic move: find the key the failure hinges on, sort the state transition spans by that key, and trace exactly when it was last written and whether that write was confirmed before the failing read happened. If state stays consistent throughout and the inputs at the failing step check out, memory is cleared, and the search moves to reasoning or the product logic layer.

Structuring the search: a layer-elimination protocol for production traces

Attribution works by elimination. Start with the layers whose evidence reads mechanically, and clear those before moving toward layers where the evidence gets more ambiguous and interpretive. Reasoning about the model's own thinking last isn't caution, it's discipline: that's the layer with the least direct evidence and the most room for guesswork, so it only gets blamed once everything with a paper trail has been ruled out.

Start with tool spans. They leave the clearest record: API error codes, argument mismatches, retry counts sitting right there in the log. If any tool span shows a failure state the workflow didn't handle, that's the attribution. No need to look further.

Move to state transition spans for memory next. Check whether the inputs at the failing step match the most recently confirmed state. A mismatch here is a memory failure, not a reasoning one.

Then check reasoning spans for workflow problems and plan drift. Look for a plan that changed between an early span and a late one with no new information arriving to justify the change. If the plan shifted for no reason, the workflow failed to enforce the sequence it was supposed to enforce.

Last, compare across runs for prompt-layer causes. If the same failure appears across multiple runs, with different inputs, different tools, different memory states, whatever's shared across those runs, and a prompt is usually the only thing that is, becomes the prime suspect. Only after tool, memory, and workflow are cleared, and a cross-run pattern points at something structural in the instructions themselves, does the prompt layer become the answer. Only after all four are cleared does the model's own reasoning become the likely culprit, since that explanation rests on the thinnest evidence and the most guesswork.