Running an ASR bakeoff for your voice AI agent

Microsoft dropped MAI-Transcribe-2-Streaming on October 1, and the streaming speech-to-text leaderboard changed shape overnight. First partial transcripts arriving a bit over 100ms after audio starts, a 2.5% final-transcript word error rate on Artificial Analysis's streaming benchmark, 60+ languages, $0.54 per audio hour as an introductory rate. Deepgram keeps adding Nova-3 coverage across languages and verticals, and ElevenLabs just doubled to a $22B valuation on enterprise voice-agent demand. Everyone running a production voice agent is quietly asking the same question: should we swap our ASR?

The honest answer is that the leaderboard doesn't know. Your agent does. "Lowest WER on Artificial Analysis" and "highest task completion on your Monday morning calls" are two very different numbers, and the second one is what your business runs on. This post is a methodology for the PM, QA lead, or voice-ops owner who has to make the swap decision and defend it afterward.

Why a leaderboard won't make the call for you

Public benchmarks are honest about what they measure. AA-WER Streaming reports word error rate paired with latency, on fixed audio sets like AA-AgentTalk, VoxPopuli-Cleaned-AA and Earnings22-Cleaned-AA, with timing measured from Silero VAD-detected end of speech. It's clean, reproducible, and useful for ranking general ASR capability.

What that benchmark doesn't carry:

• The acoustic character of your calls (narrowband PSTN, caller-side ambient noise, mobile LTE, in-car Bluetooth)
• Your domain vocabulary (medication names, SKU formats, policy numbers, member IDs, French-Canadian street names)
• Your agent's turn-taking logic, which is tuned to a specific endpoint latency distribution
• Hallucinations on silence or non-speech noise
• Code-switching mid-utterance, when a caller slips a Spanish name into English
• Rate-limit behavior under your production concurrency

And no single model leads everywhere. On AA-AgentTalk, ElevenLabs Scribe v2 Realtime leads both final and partial WER; on VoxPopuli, AssemblyAI U3 Realtime Pro leads final WER and Google's Chirp 3 Streaming leads partial; on Earnings22, Cartesia Ink-2 leads both. A model that tops aggregate WER but hallucinates on hold music, or endpoints 100ms later than you tuned barge-in for, is a regression for your product even if it's progress for the field.

What "ASR swap" actually means for a running agent

In a cascaded voice stack, ASR sits at the front of the pipeline: audio → STT → LLM → tool calls → TTS → audio. Downstream stages forgive the stage before them inside a narrow band, so a provider swap changes four things at once:

1. Transcript content. Different error distribution, different substitutions. Your prompt and intent routing were tuned to the old distribution.
2. Latency profile. Partial-transcript timing, final-transcript timing, and end-of-speech signaling all shift. Turn-taking rules fire in new places.
3. Behavior on silence and noise. Some models return nothing, some return hallucinated text, some return a short "mm-hmm" that your LLM then responds to.
4. Cost and rate-limit shape. Streaming pricing on the current leaderboard spans roughly $2 to $17 per 1,000 minutes, and that spread is a budget line at any real volume.

A swap is not a config change. It's a release, and it deserves the same gates every release gets.

The bakeoff, in five parts

1. Write pass/fail criteria before you look at a single result

The most common mistake is running the comparison first and deciding which numbers to care about second. By then everyone is backfilling the winner they already picked. Write the criteria down first:

• Primary business metric. Usually task completion or resolution on a representative call cohort. Pick a floor you will not ship below.
• Latency gates. p95 time-to-first-partial and p95 time-to-final-transcript. These feed turn-taking and perceived responsiveness.
• No-regression cohorts. Specific caller populations the new ASR has to be at least as good as the current ASR for. A 1% aggregate WER improvement that costs 3% on Indian English is a regression if a quarter of your callers are Indian English.
• Hallucination budget. Fabricated-content turns per 1,000 calls. For many agents the right number is zero.
• Compliance behavior. Does the new model still produce the exact phrases your compliance checks look for (disclosure wording, recording consent, DNC acknowledgment)?

If you can't fill the floors in before you start, you are not evaluating a swap. You're running a demo.

2. Build a representative corpus from real production

If you've been running the current agent at any volume, you already have the test set. Pull two to four weeks of production recordings, stratified so the sample looks like your real traffic:

• Caller cohorts (accent, language, device/mic quality)
• Time of day and day of week
• Call reason distribution
• The known-hard calls: ones that failed, were escalated, had corrections, triggered compliance issues

The known-hard subset matters more than the overall sample. Those are the calls where ASR differences are visible, and if the new model can't resolve them, no leaderboard delta will save you.

3. Run both ASRs end-to-end, not just transcript-to-transcript

A transcript-vs-transcript diff on recorded audio is informative but not sufficient. ASR errors behave differently once the agent reacts to them: a mis-heard street name becomes a wrong confirmation turn, which the caller corrects, which changes the latency and token budget of the whole call. Research on ASR failure modes suggests 30-40% of voice-agent "model bugs" reported by product teams are actually ASR substitution errors the LLM faithfully obeyed. You cannot see that from a WER diff. You have to run the full stack.

Two methods that work together:

• Parallel simulation over real telephony. Dial your agent with the same persona scripts under provider A and provider B, scored on the same metric suite. Not a text loopback and not a WebSocket harness; real audio over PSTN or WebRTC, because the transport is part of what you are evaluating.
• Production replay. Take real production calls and re-run them against your agent with the new ASR in place, scoring the agent's new behavior. This is the only way to know whether the swap actually moves the needle on the failures you already have.

Simulation gives breadth: hundreds of synthetic callers across your personas, in parallel, triggered from CI. Replay gives ground truth: the exact call that failed yesterday, scored under the new model today.

4. Score on audio-native metrics, not just WER

WER tells you how many words were wrong. It won't tell you whether the agent interrupted the caller, whether a silent pause before the agent responded felt too long, or whether the TTS mispronounced a policy number because the ASR captured it wrong in the first place. Those are properties of the sound of the call. A useful scoring suite for an ASR bakeoff includes at least:

• Interruption count and false barge-in rate
• Dead-air duration before agent response
• Pronunciation of named entities and numeric strings
• Caller correction turns and frustration markers
• Task completion against the stated goal
• Compliance phrase integrity

The point: an ASR swap that improves WER by 20% but widens pre-response pauses by 300ms is a worse call for your caller. Only end-to-end audio scoring will find that.

5. Decide against a scorecard, not a vibe

Fill a table like this before anyone opens Slack:

If every row isn't green or neutral, the "cheaper" or "faster" model isn't actually cheaper or faster for your product. It's an incident waiting to be filed.

When the swap is actually worth it

Even with the discipline above, there are concrete reasons to run the bakeoff and pull the trigger:

• Language coverage. MAI-Transcribe-2-Streaming supports 60 languages with automatic detection, and Deepgram's September release added Kazakh and improved models for seven more. If you're expanding into markets your current provider doesn't serve well, that's a real driver.
• Vertical vocabulary. Deepgram shipped Nova-3 Pharma built for pharmaceutical vocabulary with a focus on accurate drug-name recognition. Tuned models on domain words beat general models on the words that matter.
• Latency budget. Microsoft reports first partials in the low hundreds of milliseconds for MAI-Transcribe-2-Streaming, and Deepgram Flux benchmarks around 20ms to final transcript at the cost of higher WER (7.4%). Those are design-space tradeoffs, not drop-in wins, and the right choice depends on how aggressive your barge-in needs to be.
• Unit economics. A swap that moves per-minute spend meaningfully at projected volume is worth evaluating even if accuracy is flat.

Each of these is a reason to run the comparison. None of them, on their own, is a reason to skip it.

Where Roark fits

Roark exists for exactly this kind of question. The platform's simulation testing dials your voice agent over real phone calls (PSTN and WebRTC), scores the full call on 64+ built-in metrics with audio-native models that measure pronunciation, emotion, pace and interruptions, and turns every real production failure into a replayable regression test. For an ASR bakeoff that means:

• Build a scenario suite of personas across your real caller cohorts, with the accents, pace, emotion, and background noise your callers actually bring, in 45 languages.
• Run the suite in parallel against your agent configured with provider A and provider B, triggered automatically over HTTP so the swap can gate CI.
• Replay the production calls that failed on the old ASR against the new one, scored on the same metric suite so the comparison is apples to apples.
• File issues automatically on anything that regresses, with the call recording attached, so the scorecard effectively writes itself.

If your agent runs on Vapi, Retell, LiveKit, Pipecat, Bland or ElevenLabs, the integration is one click; the SDKs are documented at docs.roark.ai. The whole point is to make the decision on your audio, your agent, your metrics, instead of on a public leaderboard that doesn't know what your product is.

A short checklist before you start

• Pass/fail criteria written down, including per-cohort floors
• Two to four weeks of production recordings pulled, with a known-hard subset tagged
• Persona scripts cover your top 10 call reasons and your worst 10 call outcomes
• Scoring suite includes audio-native metrics, not just WER
• Compliance phrase checks in the metric set
• CI integration so the swap gates on the suite
• Rollback plan: how fast can you revert if the first week in production is worse than the bakeoff predicted?

If the answer to any of these is "we'll figure it out when we get there," you're not ready to swap. The new model will probably be fine. If it isn't, you won't know until a caller tells you.