Cost, Latency, and Quality Alerting on Production Agent Traces

A workflow loop makes the case on its own. An agent stuck re-running a step is technically "working," so latency climbs, cost accumulates with every repeated call, and no error code fires anywhere in the stack, yet the quality of what it is producing is collapsing in real time. A dashboard tuned to any one of those three signals in isolation will miss the event completely, because the failure is not expressed in any single metric but in the relationship between them. Production LLM agents generate a trace that accumulates cost, latency, and quality simultaneously, at every step, which makes splitting them into three independent dashboards a category error rather than a reasonable division of labor. The trajectory, not the individual model call, is the correct unit of analysis. Costs compound across model calls, tool invocations, and retrieval steps in sequence, while the utility of the whole exercise only becomes visible once the task is complete, and the ADE-PRF framework (Agent Delivery Engineering Predictive Reliability Framework, from Shanghai Qijing Digital Technology Co., Ltd.) found that 76% of LLM agent failures violate more than one check at once, confirming that splitting monitoring into independent dashboards is a structural mistake rather than a stylistic one. The entanglement runs in the other direction too: a latency spike caused by a slow retrieval step often forces more context into the next model call, which drives up token cost and can degrade output quality if that context has gone stale. Everything that follows in this article, from instrumentation to alert design, rests on treating cost, latency, and quality as three lenses trained on the same trace rather than three separate things to watch.

What a production agent trace must capture for alerting

The vendor-neutral standard taking shape for this work is the OpenTelemetry GenAI semantic conventions, which as of v1.41 define client spans, agent and workflow spans, MCP conventions introduced in v1.39, semantic events, metrics, and provider-specific attributes; as of June 2026 the GenAI conventions moved to their own dedicated repository (open-telemetry/semantic-conventions-genai), with the last versioned cut in the main repo at v1.42.0. Two histogram metrics form the floor beneath this work: gen_ai.client.operation.duration for latency and gen_ai.client.token.usage for token consumption, split between input and output. If those two metrics aren't present on every span, you can't reason about cost or speed at the span level, no matter how sophisticated the alerting logic layered on top.

If you adopt this standard, you need to understand a real limitation in its current maturity. As of v1.41, nearly all gen_ai. attributes carry Development stability badges rather than Stable ones, and the only Stable attributes appearing on GenAI spans are error.type, server.address, and server.port, all inherited from core conventions and none of them carrying the gen_ai. prefix. Attribute names in the GenAI conventions can change without a major version bump, so if you build a durable alerting pipeline on top of them, you need to plan for schema migration and not assume today's attribute names will still exist next quarter. This is a meaningfully different posture than treating OpenTelemetry adoption as a solved problem: the instrumentation layer is still being built in public.

MCP tracing, added in OTel v1.39, contributes mcp.method.name, mcp.session.id, and mcp.protocol.version attributes that enrich the existing execute_tool spans, and this gives traces visibility into a tool layer that had previously been a black box. None of this reaches quality, however. Quality cannot be captured by infrastructure metrics alone; it requires an evaluation signal generated at the span level, whether through automated LLM-as-judge scoring, semantic similarity measures, or faithfulness metrics, and that signal has to attach to the same trace hierarchy carrying latency and token counts. For RAG and tool-calling agents specifically, the trace needs to decompose into separate spans for retrieval, embedding, ranking, context assembly, model inference, and tool execution; a single aggregate span covering the whole session throws away the attribution information that makes any subsequent alert actionable. Harness memory state, the working memory, summaries, and artifacts the agent builds up over a trajectory, needs to be captured in the trace as structured state rather than reconstructed after the fact from prompt text, and 2026 production architecture guidance is explicit that evaluations must cover tool choice and outcomes across full trajectories, not just final answers.

Cost signals in traces and naive token accounting

Cost is the signal most teams believe they already understand, and it is the one most consistently undercounted. Token-level accounting at the API call boundary understates true agent cost, because it misses repeated context processing across a trajectory, tool fees charged outside the model call itself, and the compounding cost of recovery loops that keep a failing agent running longer than it should. The cost-utility alignment framework from Liu et al. (2608.26195) identifies three independent cost dimensions attached to every event in a trajectory: resource consumption, monetary cost, and risk. Token accounting at the call level captures only part of the monetary dimension and none of the risk dimension. A cost dashboard built purely on token counts is measuring a fraction of what it claims to measure.

Repeated context processing is the dominant driver of this undercount in long-horizon agents. The same context gets reprocessed on every model call within a trajectory, so a multi-step trajectory carrying a substantial shared context accumulates far more token spend in aggregate than the same number of calls would if they were isolated from one another. Tool call fees compound the problem from a different angle: external API charges incurred per invocation are invisible to token accounting entirely, yet they appear in trace spans, and without MCP-layer span data attached to the trace, those fees cannot be traced back to the workflow step that actually caused them.

The practical consequence is in where the alert threshold gets set. A loop can stay within its per-call limit while running dozens of unnecessary iterations end to end, so a per-call token budget alert misses it, because no single call ever trips the threshold. But a per-trajectory cost budget alert catches that failure, because it sums spend across the whole trajectory instead of checking each call on its own. Instrumentation needs to roll token counts up by session, feature, and user, and attach those rollups to the same trace hierarchy carrying tool spans, so that when an alert fires, the spend can be attributed back to the specific workflow step responsible. The same loops and the same stale context that inflate cost also drive latency out of bounds, as the next section shows.

Latency signals that matter at the span level and the tail-percentile problem

Average session latency is as misleading for speed as aggregate token counts are for cost, and for the same underlying reason: it collapses a trajectory made of structurally different spans into a single number that obscures which part of the system is actually slow. Span-level decomposition is what reveals which layer owns the latency, whether retrieval, embedding, ranking, model inference, or tool execution, and a session-level average cannot distinguish a slow retrieval step from a slow model call or an unbounded tool loop. Context length acts as a latency lever hidden inside what looks like a cost problem: long contexts raise both token spend and inference time together, so a trajectory accumulating stale context degrades on both dimensions at once. That correlation is detectable by comparing gen_ai.client.token.usage for input tokens against gen_ai.client.operation.duration across spans in the same trajectory.

Tool call latency introduces a kind of variability that model-side monitoring was never built to see. MCP spans capture this I/O variability directly, and bounding the delay of an individual tool call requires isolating it as its own named span rather than letting it get absorbed into an aggregate session time where it disappears from view. Time-to-first-token is a distinct metric from total generation time, because users experience the delay before the first token arrives rather than the delay before the full response completes, so an agent that streams early can tolerate a higher total latency without any drop in perceived performance.

The threshold design that follows from all this sets P95 on a per-span-type basis, retrieval, tool, and inference tracked separately, rather than on the session average. A retrieval span sitting consistently at P95 points to a retrieval problem specifically, while a model inference span at P95 alongside normal retrieval times points to a different problem requiring a different fix entirely. The ADE-PRF framework shows that infrastructure-level metrics such as process liveness, CPU, and memory can stay perfectly normal even as semantic quality degrades underneath them, which matters for how these alerts get read. Latency alerts, in other words, need to be read alongside quality signals rather than treated as an independent axis, because a system can be running fast and clean by every infrastructure measure while the actual work it is producing falls apart.

Quality signals in production traces and their silent degradation

Quality regressions are the most dangerous failure mode in a production agent precisely because they produce nothing for infrastructure monitoring to catch. There is no error code for an agent that completes a task plausibly but wrongly, and the degradation often unfolds over days of a judge score drifting downward before anyone watching a conventional dashboard notices anything is wrong. Context rot is one of the clearest mechanisms: quality degrades as context accumulates stale or redundant information across many steps, even though the model still has the right tools and the right prompt in front of it. The signal-to-noise ratio inside its context has degraded so far that the right inputs can no longer produce the right output.

Prompt drift compounds across steps rather than announcing itself at any one of them. A small output change at step N can cascade into failures at steps N+1 through N+5, and you can see that cascade only by tracing the full trajectory, not by inspecting any single call. Tool schema drift is subtler still: an argument name change, say from customer_id to account_id, produces no stack trace whatsoever, yet task completion drops, retries increase, and outputs become confidently wrong rather than visibly broken. The ToolScan taxonomy, from an ICLR 2025 workshop, catalogs seven recurring tool-use error types covering insufficient API calls, incorrect argument values, incorrect argument names, incorrect argument types, repeated API calls, hallucinated function names, and output or format errors.

To catch any of this you need to instrument quality as a scored signal attached directly to trace spans, using LLM-as-judge metrics for faithfulness, relevance, and safety applied at the step level, not only at the session boundary. Evaluation, on this view, is the core of observability rather than a post-hoc annotation layered on afterward. Multi-agent systems raise the stakes further: analysis of LLM API traffic shows multi-agent failure rates spanning a wide range, driven largely by spec ambiguity between agents and coordination failures between them, and most of these failures are invisible to monitoring operating at the API layer. A judge score sliding downward over the course of a week is fundamentally a trend to be detected, not a single threshold to be breached, and any alerting infrastructure built for this domain has to support rolling trend detection against a baseline alongside ordinary point-in-time threshold alerts.

Setting alert thresholds across cost, latency, and quality

Pairing a per-trajectory cost threshold with a quality trend check before escalating anything resolves the most common false alarm in this space: a cost spike with no accompanying quality drop is often just a legitimately complex task taking longer and costing more to complete properly, while a cost spike arriving alongside a quality drop signals a loop or a context-bloat failure that needs attention immediately. The same logic connects latency to quality directly. A retrieval span consistently hitting P95 latency while judge scores on retrieval-grounded responses decline together describes a retrieval quality problem, not a model problem, and the alert generated from that pattern should route to the retrieval layer rather than triggering a model swap that wouldn't address the actual cause.

Tool-call frequency serves as a third composite signal worth tracking directly rather than inferring after the fact. If you count tool invocations per trajectory, an abnormally high number of calls to the same tool within a single trajectory is visible as a loop indicator, and that same loop produces both a cost anomaly and a latency anomaly before quality has fully collapsed, so watching tool-call counts alongside cost and latency catches the failure earlier than waiting for quality scores to confirm it.

To build this into working alert logic, you need a few distinct threshold types operating together, not one master number. Set static thresholds on per-span latency, at P95 by span type, and on per-trajectory token budget, so you catch the clearest boundary violations in speed and spend. Put trend alerts on rolling judge score averages, flagging a sustained decline over a configurable window instead of reacting to any single low-scoring trace, because a single bad trace is often just noise, while a sustained decline is the real signature of silent quality degradation. Rate alerts belong on tool-use error categories drawn from the ToolScan taxonomy, particularly incorrect argument names and hallucinated function names, both of which appear in trace spans as structured errors that can be counted reliably across a deployment window. Anomaly detection belongs on multi-agent coordination spans specifically, so you watch for the point where cascade failure rates begin to climb across agents, not within any single agent's trajectory.