LLM App Framework Impact on Traceability

Most agent failures in production trace back to the scaffolding around the model. A review of agent architecture out of Shanghai Jiao Tong University frames this as a historical shift: capability that once lived inside model weights has moved outward, into memory stores, skill registries, interaction protocols, and the harness code that ties them together. That matters because it relocates the question an engineering team has to ask when something breaks. The model did what it was asked to do with the inputs it received; the failure lies in what the harness handed it, or in what the harness did with what it returned.

That relocation changes what debugging means in practice. If reliability now lives in the runtime infrastructure around the model rather than in the model's weights, then debugging a production incident is a matter of finding which layer of that infrastructure, prompt, tool, memory, or workflow logic, produced the bad outcome. A team that can't see inside those layers at the moment of failure has no path to the root cause, no matter how capable the underlying model is. The rest of this piece works from that premise: the framework a team picks to build its agents decides, often before a single line of application code is written, how much of that infrastructure is visible when something goes wrong.

What a trace needs to capture for failure diagnosis

A trace that records only the prompt going in and the completion coming out tells you almost nothing useful about why an agent failed. Agent diagnosis requires a trace that links every retrieval, every tool call, every memory read and write, and every reasoning branch back to whatever caused it, across the full run from the user's request to the agent's final answer.

A survey of evidence tracing and execution provenance lays out the specific pieces a useful agent trace has to contain: retrieved evidence, tool outputs, memory items, observations, intermediate claims, actions, messages passed between agents, and the final answer, all connected by relations that show what caused what. The survey breaks this down further into the sources a trace draws from, the units it records as evidence or execution, the relations that connect those units, the granularity and timing of the capture, how the trace is represented, and the trust functions it supports. Each of those is a place where a framework can quietly drop the signal a team will need later.

The gap between what a single LLM call looks like and what an agent run looks like is the gap between structural tracing and semantic tracing, and all three of those failures are costly, invisible to span health monitoring alone.

For a single LLM call, the unit worth watching is the call. For an agent, the unit that matters is the trajectory, the full sequence of steps from the user's request to the point where the agent declares the task done. A framework that has no concept of a trajectory as a thing to be modeled gives a team no way to detect that an agent is drifting across ten steps even if each of those ten steps looks fine in isolation. The provenance survey points to TRAIL, which uses annotated agent traces to localize the step, component, or interaction responsible for a given failure, and to MAST, which found that multi-agent failures come from specification and system design problems, misalignment between agents, and weak task verification. A trace that only records single calls cannot capture any of those causes. Taken together, this sets a working checklist for what a trace has to carry: tool calls recorded step by step with their arguments and return values, memory reads and writes tied to the steps that triggered them, messages between subagents when more than one agent is involved, a causal link between a retrieved document and the claim it fed, and metadata that ties individual spans back to the user goal the whole trajectory was serving.

How OpenTelemetry's GenAI conventions set a common floor

OpenTelemetry's GenAI semantic conventions give the industry a shared vocabulary for describing what happens inside an LLM call, and frameworks that adopt it produce traces that move cleanly between observability backends. That standardization is real and useful, but it sets a floor, not a ceiling. What a framework builds on top of that floor still varies enormously from one to the next.

OpenTelemetry's own guide to AI observability describes how the floor works in an agent context: each tool call, each LLM invocation, and each retrieval step becomes a child span, which together produce a full trace of the reasoning chain. Standard gen_ai.* attributes capture input and output token counts, latency, and errors. The conventions also define a small set of GenAI event types: gen_ai.client.inference.operation.details, gen_ai.evaluation.result, and gen_ai.client.operation.exception, each carrying its own defined attributes. That's what a framework gets simply by emitting standard OTel traces.

What the conventions leave out is just as telling as what they cover. They don't standardize semantic labels on trajectory state, so nothing in the spec tells a backend whether an agent is looping or off-task. They don't standardize memory provenance, cross-agent causal links, or claim-level attribution of evidence to the answer it supported. Those gaps are exactly where frameworks start to diverge from one another, and the next two sections take up what fills them and what doesn't.

OTel-compatible tracing has become the standard target for agent runtimes that ship their own observability hooks. A framework that exposes clean hooks for that manual work gives a team real control over what ends up in the trace. One that doesn't leaves that work to whoever is willing to fork the SDK.

What framework architecture decides about default instrumentation

A framework's internal architecture, how it models tool calls, memory, handoffs between subagents, and reasoning steps, decides directly which of those pieces produce traceable spans and which stay opaque. A framework can only have its runtime events instrumented if it first surfaces them as distinct, nameable things.

The externalization review lays out six dimensions along which a harness externalizes capability that used to live in the model: three modules, Memory (state persistence, failure recording, cross-session context), Skills (reusable routines, staged loading, failure-driven revision), and Protocols (deterministic interfaces, structured invocation, schema contracts), and three operational surfaces, Permission (sandboxing, filesystem isolation, network restrictions), Control (recursion bounds, cost ceilings, timeout), and Observability (structured logging, execution traces, aggregate metrics). These six are coupled dimensions, and instrumenting one does not account for the others. Instrumenting Observability alone does nothing for a failure that originates in Memory or Control, because the trace still won't record the event that actually went wrong.

Typed tool interfaces give a framework a natural place to hang a span. When tool arguments are validated against a Pydantic model or a JSON schema before the call goes out, that validation happens at a defined boundary in the code, and a framework built around that boundary can emit a span recording both the call it intended to make and any validation failure along the way. A framework built around untyped tool calls, arguments passed as loose dictionaries with no schema behind them, has no such boundary to instrument, and that signal is lost before it's ever generated.

Memory architecture works the same way. The provenance survey calls this out directly: memory lineage is its own distinct provenance relation, and it has to be modeled explicitly to exist in the trace.

Workflow control logic tends to be the least visible of the three. When a run fails because of a runaway loop or a bad routing decision, the trace contains an abnormally long sequence of child spans that each look perfectly healthy on their own. Nothing marks the loop. Nothing marks the branch that sent execution down the wrong path. Two frameworks can both emit fully OTel-compatible traces and still produce wildly different failure visibility, since one names memory retrieval, tool validation, and subagent handoffs as distinct spans while the other folds all of it into a single LLM-invocation span that tells you a call happened and little else.

Why structural traces stay green while semantic failures accumulate

The failures that do the most damage in production, looping, drifting off task, tool schema mismatches, prompt ambiguity that compounds turn over turn, fail to register in structural traces. Whether a framework can catch them comes down to whether it was built to emit semantic signals alongside the structural ones.

A paper on the ADE-PRF framework documents what it calls a "false prosperity" phenomenon: external observability metrics stay normal for extended stretches of production operation, a stretch in which the actual quality of what the agent produces quietly degrades. The paper's own 15-day production deployment, run across six different agent profiles, confirmed that this kind of degradation can occur for real while every external metric a team is watching continues to look fine.

Tool schema drift is a version of this problem that lives squarely at the framework level. When a tool's underlying API changes in production but the framework's registry of that tool's schema doesn't get updated to match, the result is a mismatch: calls go out malformed, and they come back either as outright errors or, worse, as results that look fine but are silently wrong. A framework that versions its tool schemas and diffs them against the live API at the moment of invocation can turn that mismatch into a traceable event the moment it happens. A framework that just passes arguments straight through has no way to notice.

Prompt ambiguity is harder still, because it compounds across turns in a way no single span can show. Each completion in the sequence can look syntactically fine on its own, while the agent is, turn by turn, settling deeper into a suboptimal reading of an instruction that was underspecified from the start. Catching that requires semantic labeling applied at the level of the whole trajectory. Per-span error codes have nothing to say about it, because no individual span is wrong.

Silent drift in the underlying model raises a related problem. A framework that periodically replays a set of known-good traces against current production can catch a behavioral change introduced by an upstream model update before it does damage. A framework with no such replay mechanism has no way to know that the behavior of its own evaluators has quietly shifted. The general risk stands regardless of whether any one incident gets definitively attributed.

The framework-level answer to all of this is per-turn semantic labeling: flags like is_agent_looping, is_off_task, or tool_call_wrong attached to each turn of the trajectory, giving downstream analysis a signal that no structural span can provide on its own. Frameworks that expose hooks for attaching labels like these to spans make this kind of detection tractable. Frameworks that don't leave a team building an entire separate analysis pipeline outside the trace just to answer questions the trace should have already answered.

How root cause attribution needs granularity most frameworks skip

Knowing that a run failed and knowing which layer caused it are two different problems, and the granularity needed to answer the second one has to be built into the trace at the moment of capture. It can't be reconstructed later from a trace that wasn't built to hold it.

The provenance survey treats root cause attribution as the function that turns a bare failure signal into something a team can actually act on: the fault gets assigned to the component responsible for it, whether that's the model, the harness, the environment, or the grader, and that assignment tells the team where to intervene. If a team skips that attribution step, it risks fixing a component that was never the problem; the change it ships leaves the real cause untouched and waiting to fail again.

TRAIL, the tool cited in the provenance survey, works by using annotated agent traces to localize the specific step, component, or interaction behind a failure. Doing that requires a trace detailed enough to tell a retrieval failure, the wrong document got pulled, apart from a reasoning failure, the right document got pulled but the agent drew the wrong conclusion from it, apart from a tool failure, the conclusion was right but the resulting call came out malformed. Three different causes, three different fixes, and a shallow trace can't tell them apart.

A framework that flattens a multi-step run into a shallow tree of spans makes this worse by obscuring the distance between where something first went wrong and where it finally surfaced as a visible error.

Each of the five harness layers needs its own kind of evidence to be attributable. Prompt failures appear in the reasoning steps that follow an instruction that was underspecified to begin with. Tool failures appear in the return value or the exception thrown by a tool-call span. Memory failures appear in the gap between what was actually retrieved and what the reasoning that followed assumed had been retrieved. Skill failures appear in the output of whatever packaged routine was called. Workflow failures appear in the branching or termination logic that decided what happened next. A framework that collapses all five into one undifferentiated LLM-invocation span makes attribution at this level impossible, because the trace no longer preserves the boundaries between them.

Multi-agent systems raise the difficulty further. A framework that drops context at the boundary between agents has already made that kind of attribution impossible, no matter how well it instruments what happens inside each agent on its own.