October produced a wave of new voice stacks worth taking seriously. Block's Dessa landed on October 23 as a full-duplex speech-to-speech model that laughs, pauses, and interrupts the way people do. The week before, Amazon pushed Nova 2.5 Sonic to GA. Mistral shipped Voxtral 3 and a voice-agent SDK on October 22. Decagon put Voice 3 and Chord into production on October 1, and Microsoft dropped MAI-Transcribe-2-Streaming the same day. If you run a production voice agent, you probably have a Slack thread this week from someone asking when you plan to try one of these.
That is where product managers quietly get into trouble. The gap between a demo that sounds human and a stack that handles your Tuesday morning traffic is still measured in weeks of rework, not minutes of listening. Shadow mode is the pattern that closes that gap without betting your call quality on vibes.
What shadow mode means for a voice agent
Shadow mode in software is old news: route the same real request through the new system in parallel with the current one, keep the current system in charge of the outcome, and compare. One recent write-up puts it bluntly as "grade the new version on real traffic before it serves anyone." It is cheap, it is boring, and almost no agent team uses it.
For a voice agent the pattern needs a translation, because a phone call is not an HTTP request you can fork into two pipelines. There is one caller, one audio stream, and one microphone that gets to speak. You cannot have the AI and the human both answer at the same time without turning the call into a mess. So shadow mode for voice means picking one of three parallel-run shapes that preserve the live call while still exposing the new agent to it.
Done well, this gets you something no amount of staging gets you. One deployment guide frames the problem precisely: an agent that hits 94% intent recognition in UAT may hit 81% on day one of live calls, not because it was built badly but because the real-call distribution is wider than any test suite captures. Shadow mode is how you discover that delta before a caller does.
Three shapes of voice-agent shadow mode
1. Passive scoring (silent AI)
The simplest version. The AI listens to the live call in real time and generates the response it would have given, but never speaks. Audio goes to a sidecar that transcribes, runs the agent's logic, and logs the proposed response, the proposed tool calls, and the latency it would have taken to produce them. The human agent continues as normal.
This is what the AgentCenter team describes when they talk about "observing, not acting." You get real-world evidence of how the agent behaves under real audio conditions, and a direct comparison against what the current process produced. Good first stage, especially for inbound support where the human handling the call is already competent and the risk of parallel-run interference is zero.
The limitation is that you never actually hear the AI talk. Pronunciation, pacing, barge-in handling, and the way the TTS lands on an upset caller are all missing from the data. You find out what the agent would have said, not how it would have sounded. For most PM teams, that is enough to start, but it is not enough to ship.
2. Scenario replay (simulated shadow)
The sharper version, and the one most teams end up with once they have passive scoring working. Every real call that comes in gets captured, and after it ends you replay the same scenario against the new agent in simulation: same caller profile, same language and accent, same background noise, same intent, same tool-call context. The new agent handles the whole conversation end to end over a real telephony leg, and you score it the same way you score live calls.
This is where shadow mode stops being a mirror of production and starts being a regression harness that grows with your call volume. Every weird call that happens in the real world becomes a scenario the new stack has to pass before it earns traffic. It also lets you shadow a stack you cannot physically plug into the live flow yet, which matters when the thing you are evaluating is a different platform or a different model vendor rather than a different prompt.
3. Whisper mode (AI drafts, human reads)
The riskiest of the three to run cleanly, but the one that produces the strongest signal when it works. The AI listens live and generates a response, and the human agent sees the proposed response on screen with the option to use it, edit it, or ignore it. The human is still in charge, but every turn gives you a concrete artifact: did the human pick the AI's draft, modify it, or write their own, and why.
Whisper mode is awkward because it changes the human's job while you are running the experiment, which contaminates your baseline. It is also where you learn the hard truths about tool-call latency, because the AI's draft is only useful if it is on screen before the human has started typing. Save this one for later, and only on a cohort of agents who know they are in the study.
What to measure in shadow
Shadow mode is only as good as the comparison you draw. A log of AI responses that nobody grades is just storage cost. These are the metrics that have earned their place in a voice-agent shadow program, grouped by what question they answer.
Would the AI have said something you can live with?
- Response agreement rate against the human: at the intent level, at the tool-call level, and at the final-answer level. Expect divergence to be high at first and to drop as you fix prompt and retrieval issues.
- Dangerous-content delta: cases where the AI's draft would have quoted a wrong price, promised a wrong SLA, or committed to a wrong appointment slot. This is the number that gets you fired. Grade it manually at first.
- Compliance behavior: disclosures, consent capture, state-specific scripts. For regulated verticals the shadow log is also your audit trail that the agent would not have violated policy before you let it near a caller.
Would the AI have sounded like a system you want on the phone?
- End-to-end latency from caller stop to AI start, including the LLM, any tool round-trips, and TTS. The dilr.ai writeup notes that agents designed around sub-400ms tool-call round-trips can encounter 1,200ms responses in production they were never built to handle gracefully. Shadow mode is where you learn whether yours is one of them.
- Interruption and overlap behavior: how often the AI would have barged in on the caller, and how often it would have missed a legitimate interruption. Only measurable in shapes 2 and 3, since passive scoring has no audio to analyze.
- Pronunciation on your actual proper nouns, SKUs, and street names. The ones your demo got right because they appeared in the prompt.
Would the AI have cost you a call?
- Escalation point delta: at what turn would the AI have handed off, versus at what turn did the human resolve.
- Average handle time delta, with the usual caveats that fast is not the same as good.
- Caller sentiment proxy at the point the AI would have taken over. Not perfect, but useful as a direction indicator.
Pick five or six of these and write the thresholds down before you start. Shadow data that nobody has a prior opinion about becomes an argument, not a decision.
Exit criteria: when do you leave shadow
Shadow mode is a stage, not a lifestyle. The point is to graduate to canary and then to real traffic, and that requires knowing in advance what "ready" looks like. A reasonable staged ramp, borrowing from one published deployment plan, looks like silent monitoring, then shadow mode, then a pilot at ten percent of calls, then expansion in roughly thirty-percent steps.
The useful discipline is to attach numeric gates to each transition, not calendar dates. For example:
- Agreement rate above X% on the top ten intents, measured over at least a thousand shadowed calls, with no intent below Y%.
- Zero dangerous-content incidents over the last N shadowed calls.
- P95 end-to-end latency under 1,200ms on your actual telephony path, not on a benchmark harness.
- No open critical issues against the metrics you promised compliance.
A gate that is a date tells you nothing. A gate that is a number tells you whether you learned anything in the last two weeks. If you do not hit the gate, you stay in shadow and fix the thing that is failing. If you hit the gate on every metric, you move to canary with a tight rollback plan and a monitoring dashboard that someone owns.
Failure modes to design around
A few patterns show up in almost every shadow program that stalls.
The "it was fine in shadow" rollout. The AI performed well on the hundred calls someone sampled by hand, and badly on the ninety-nine thousand that nobody looked at. If you are not scoring every shadowed call automatically, you are not running shadow mode, you are running spot checks.
Drift between shadow and canary audio. The shadow stack ran against a cleaned-up recording of each call. The canary stack ran against live PSTN audio with jitter, codec artifacts, and the caller's air conditioner. If the shadow leg does not use the same audio path as production, the numbers do not transfer.
Comparing to the wrong baseline. Your humans are not perfect, and some of the "divergence" between the AI and the human is the human being wrong. Pick a small sample weekly where a reviewer grades the human too, so you know whether an eighty-percent agreement rate is a problem with the AI or a feature of the humans.
Running shadow forever. Shadow mode is cheap to run and comforting to look at, so teams let it run for months while nobody pulls the trigger on canary. The gates exist to force the decision.
Where simulation does the heavy lifting
Shadow mode on a voice agent is unglamorous infrastructure: audio capture, scenario reconstruction, replay harness, metric suite, dashboards, issue filing. Building it in-house is a quarter of engineering effort before you have made a single product decision. This is where Roark fits.
Roark's simulation testing dials voice agents over real phone calls (PSTN and WebRTC), driven by personas with distinct voices, accents, speech pace, emotional register, and background-noise environments. That is the engine behind shape two: for every real call you capture, you can replay the same scenario against the candidate agent over a real telephony leg, score it on the same metric suite, and compare. The runs are triggered over HTTP so they can gate CI, which is how you keep shadow data flowing without a human babysitting it.
The other half is observability on the live side. Roark scores every production call against 64+ built-in metrics plus any custom ones you define, using audio-native models that grade pronunciation, emotion, vocal stress, pace and pauses, and interruptions rather than only the transcript. Failures get filed as issues automatically, which gives your shadow program a clean definition of "something to investigate" instead of a wall of raw calls. And because production call replay captures real calls and lets you re-run them against updated agent logic, every real failure becomes a regression test the next candidate has to pass. The shadow log stops being a one-off dataset and becomes a growing bar.
None of this removes the work of defining your gates, writing the thresholds down, and holding the line on canary graduation. It does remove the quarter of engineering effort you would otherwise spend building the harness, so the PM gets to spend their energy on the thing that matters: whether this new stack is actually ready for the mic.
