Voice and Browser Agent Observability Gaps
Standard tracing misses where voice and browser agents actually fail.
An engineer spends a day chasing what looks like a reasoning bug: the agent canceled an appointment nobody asked to cancel. The trace shows a clean prompt and a clean completion, so the model gets blamed, when the actual fault sat upstream in the speech recognizer that handed the model the wrong words to reason over. That gap between what the trace shows and where the fault lives is the subject of this piece, and it holds for both voice agents and browser agents.
Failure modes standard LLM tracing misses in voice and browser agents
Most observability tooling built for LLM agents watches one layer: the prompt going in and the completion coming out. That's a reasonable design for a text chatbot, where the model is close to the entire system. Voice and browser agents add stages before and after the model that a prompt-and-completion trace never touches, and those stages are where a large share of real failures start. Voice agents place a speech recognizer, a turn-taking system, and a speech synthesizer around the model, and each one has its own way of silently feeding the model bad input or mangling good output. Browser agents operate inside a web page whose structure changes underneath them, often with no relationship to anything the model reasoned about. Either way, a failure born in ASR transcription or in a shifted DOM selector is indistinguishable, in a standard trace, from a model reasoning error, so teams end up debugging prompts when they should be debugging a microphone feed or a page layout. This isn't a theoretical gap confined to prototypes. Production stacks built on LiveKit, Pipecat, Vapi, and custom WebRTC setups now run healthcare scheduling, collections calls, sales qualification, and support routing, and the volume running through them (Vapi alone reports more than 62 million calls a month) means these invisible failure modes are already happening at scale, not waiting to happen.
Silent propagation of ASR corruption and turn-detection errors through voice agent layers
Automatic speech recognition errors rank among the most dangerous failures a voice agent can have, because the transcript the model receives is already wrong by the time the model starts reasoning. Every step after that, the tool call, the spoken reply, builds on a corrupted foundation, and the model has no mechanism to flag that its input was unreliable. The failure mode plays out concretely: a caller says "move my cardiology appointment to Friday," the speech recognizer hears "cancel my cardiology appointment," the model reasons correctly over the transcript it was given, the scheduling tool executes a destructive cancellation, and the voice synthesizer reads back a confident confirmation. Everything downstream of the ASR error worked exactly as designed. The incident still reads, to anyone reviewing it after the fact, as a tool bug or a model misfire, when the actual fault happened before the model ever saw a word.
Turn-detection has its own failure class, separate from transcription accuracy. Barge-in detection that fires late causes the agent to talk over the caller. End-of-turn detection that runs slow pushes latency out directly: P99 time-to-first-audio can more than double, reaching 1.4 seconds after a provider change, and a model-layer trace attributes nothing to that regression because the model was never involved in the delay. Audio quality problems, jitter, packet loss, codec switching mid-call, degrade what the caller experiences even when the model performs flawlessly, and standard HTTP-trace APM tooling has no visibility into any of it.
Newer speech-to-speech models make this harder to catch, not easier. Models like GPT-Realtime-2 and Gemini 3.8 Live fold what used to be three separate, inspectable stages, speech-to-text, language model, text-to-speech, into one opaque processing unit. The internal state that stage-level spans used to expose isn't there to inspect anymore. That makes quality evaluation on the final audio output more important, not less, since it may be the only place left to catch what the collapsed architecture hides. And some of the worst versions of this failure stay invisible until they're large: low-confidence ASR on a specific accent cohort or a specific channel type degrades quietly, invisible in aggregate task-completion numbers, until enough calls have failed that the pattern becomes visible in the data.
Required contents of a complete, attributable voice agent trace
Fixing attribution starts with the shape of the trace itself. A voice agent needs one root span per call, with child spans underneath it for speech recognition, model reasoning, tool execution, and speech synthesis. A cleaned-up transcript sitting next to a model completion log doesn't give you that structure, no matter how readable it looks.
The span tree for a single call's detail view needs to expose, as named and independently inspectable nodes: the raw audio input, the ASR transcription along with its confidence score and language tag, turn-detection events covering endpointing and barge-in windows, the model's reasoning step tagged with its prompt version, each tool call with its arguments and results, the TTS synthesis step with its latency, and a call-level outcome span tying the whole sequence together.
Keeping the assistant's audio and the customer's audio as two distinct, synchronized channels, rather than mixing them down into one mono track, is the detail that makes barge-in debugging possible. If the customer's channel shows speech starting at t=1.4s while the assistant's channel shows text-to-speech still playing at t=1.6s, that's a 200-millisecond barge-in detection gap, and it is completely invisible in a transcript that only records words, not timing across channels.
Evaluation scores need to attach at the level of the individual turn, not the call as a whole, and need to cover two separate things: transcript quality, meaning ASR accuracy measured against a reference, and audio quality, covering MOS, clipping, silence, and noise. Scoring both at the turn level is what lets a failure get assigned to the stage that actually caused it instead of defaulting, by habit, to the model. The same structure serves compliance review: a team cannot prove whether consent language was heard, redacted, retained, or skipped if the audit artifact is missing raw audio and stage-level spans, and that's a regulatory exposure as much as an engineering one.
Fleet-level alerting needs to match the kind of metric it's watching. A static threshold fits a hard SLA like a server returning a 5xx error. Anomaly detection fits a metric that naturally moves with traffic, like turn latency during peak hours or word-error-rate shifting under a new accent mix in the caller population. Treating both metric types the same way produces either alert fatigue or blind spots, depending on which one gets the wrong treatment.
DOM-state drift as a distinct class of browser agent failure
Browser agents fail for a different structural reason than voice agents do. Where voice failures tend to originate in degraded or misread audio, browser agent failures tend to originate in an environment that changed while the agent wasn't looking. A planning step picks a selector on Tuesday that works fine. A retailer renames an attribute on Wednesday, the same selector now matches nothing, or worse, matches the wrong element on the page, and the agent proceeds as though the action succeeded. The model didn't reason badly. The page it was reasoning about simply isn't the page it was trained or tested against anymore.
A strong benchmark score offers no protection against this. A browser agent that scores well on a fixed benchmark can still produce much lower task completion once it's running in production, and the drop has nothing to do with model regression. It's the gap between a frozen benchmark, static by construction, and a live page where the DOM, modal dialogs, session state, and rate limits all keep shifting. Modals that pop up unexpectedly, sessions that expire mid-task, and endpoints that start rate-limiting requests each create their own silent failure mode. An agent operating on an expired session keeps working from stale state with nothing telling it that state is no longer valid. An agent that hits a rate limit may have no way to tell a retriable error from a terminal one, and without that distinction it can either give up too early or retry into a wall.
The stakes here run higher than they do for voice. A mis-transcribed appointment can be corrected with a follow-up call. A submitted order, a transferred payment, or a deleted record often has no undo. Silent misattribution in a browser agent carries consequences that silent misattribution in a voice agent does not, and that asymmetry is reason enough to treat the two failure surfaces as genuinely different engineering problems.
Collapsed attribution in multi-agent handoffs and multi-modal traces
Attribution gets harder once agents start handing work off to each other. When a specialist agent fails after taking a handoff from a voice agent, the usual response is to look at the specialist in isolation, but that view hides everything that came before: the quality of the transcript it inherited, the tool results it was already given, the memory state already written before it took over. None of that context travels with the failure unless the trace was built to carry it across the handoff boundary.
Speech-to-speech models make the same problem worse at the level of a single call. Once speech recognition, reasoning, and synthesis are fused into one opaque unit, there's no way to decompose the hidden state inside that unit by stage. So when a handoff happens right after a speech-to-speech turn, the agent receiving that handoff inherits context with no clear record of where it actually came from.
Root-cause attribution is already a hard search problem within one agent's own trace, where a failure has to be traced back through model, harness, environment, or evaluator to find which one actually caused it. Across a handoff boundary, that search gets harder again, because each agent was typically instrumented on its own, with no shared parent span connecting their two executions into one record.
One specific failure disappears almost entirely at the handoff boundary: the workflow loop. Inside a single agent's trace, a loop where the agent keeps retrying the same failing action is a recognizable, traceable pattern. Spread across two agents, an orchestrator that keeps re-delegating to a specialist because the specialist keeps failing looks, from the outside, like two healthy agents each making one ordinary call, because the trace structure that would reveal the loop was never built to span both agents at once.
Treating each modality as a distinct instrumentation layer with its own signal vocabulary
The fix isn't adding a few audio fields to an existing LLM trace schema. Voice and browser agents need span structures and evaluation rubrics built around their actual failure surfaces from the start.
For voice, the signal vocabulary that needs to live on the span, as a first-class attribute rather than a log annotation tucked in afterward, includes: ASR accuracy measured as word-error-rate against a reference or known utterance, an audio quality score covering MOS, clipping, silence, and noise, turn-detection events spanning endpointing, barge-in windows, and double-talk, TTS synthesis latency, and a call-level task-completion outcome.
Audio files belong in object storage, with only the URL attached to the span as an attribute, not the raw audio sitting in the span's own attribute table, because keeping the raw audio out of the span keeps the trace lightweight enough to query and store at scale. An OTLP collector struggles under large payloads, and base64-encoded audio inflates storage cost fast. Getting this piece of the design wrong doesn't just raise a bill, it can make the system too slow or too expensive to keep the very traces that make failures debuggable.
For browser agents, the equivalent vocabulary centers on an entirely different set of signals: DOM-state snapshots taken at each action boundary, selector-resolution results recorded as matched, unmatched, or matched-but-hidden, session-state validity checks, and verification of whether an action's outcome actually matches what was intended. None of these have any analogue in a standard LLM span schema, because none of them exist in a text-only agent's world.
Harness failures, prompt ambiguity, tool schema drift, workflow state silently mutated mid-run, make up a meaningful share of production agent failures generally, and they're easy to miss because the harness fails through quiet state changes rather than loud errors: context truncation that drops a critical instruction, a retry that re-enters a prompt with subtly different state attached, a sampling override applied per-tenant that nobody wrote down anywhere. Catching these requires span-level visibility into the harness itself, not just into the model sitting inside it.
All of this needs to live across six distinct classes of data: call metadata, audio, transcripts, spans, evaluation scores, and tags. Each class carries its own retention rules, its own access controls, and its own compliance requirements, and each deserves to be designed as its own store, even in a system where one platform happens to manage all six.
Replay validation and continual learning as the mechanism for closing the production gap
Capturing the right spans solves visibility. It does not, on its own, solve reliability. A team that can finally see that a confidence drop in ASR caused a bad tool call still needs a repeatable way to turn that observation into a fix that's actually been checked, not just one that seems plausible on a second read.
The way to close that gap: every failure found in production, an ASR-corrupted tool call, a selector miss caused by DOM drift, a barge-in latency breach, becomes a benchmark case going forward. Once a specific production trace is required to pass before the next version of the agent ships, the same bug can't ship a second time under a different disguise.
The loop runs in both directions. Production traffic surfaces failures that no offline test set ever contained, a particular accent cohort, a particular version of a page's DOM, a particular sequence of modal dialogs, and once one of those failures is caught, it joins the fixed evaluation set that every future version has to clear before release. The test set keeps growing to match what the real world has already thrown at the system.
For voice agents specifically, the most useful signals frequently appear in production traffic rather than in any rubric written ahead of time: a regression in one accent cohort, a sudden spike in barge-in failures right after a TTS provider switch, an ASR accuracy drop tied to a new device type entering the call mix. Catching those depends on surfacing them automatically out of production traffic, because no predefined checklist was ever going to anticipate them in advance.
