The last few months have been good for anyone who ever tried to argue that raw Word Error Rate was a bad way to grade a voice agent's ears. Pipecat's open-source STT benchmark scores every provider on semantic WER (does the meaning survive?) and time to final segment (how long after the caller stops does the last transcript arrive?), and every major STT vendor has now published their standing on that chart. AssemblyAI is bragging about Universal-3.5 Pro's spot on the Pareto frontier at 1.22% semantic WER and 282 ms TTFS median. Soniox is claiming 1.25% semantic WER at 249 ms TTFS median. Speechmatics has a post explaining exactly why they're happy where they sit on the Pipecat chart. Deepgram, Nvidia, Cartesia, all in the leaderboard.
That's genuine progress. WER counting "gonna" vs "going to" as two errors was always nonsense for a downstream LLM. But if you're the engineer wiring this thing into a phone number your users call, none of these leaderboards actually answer the question you have. They tell you whether an STT model is fast and accurate on 1,000 short clips from a public dataset. They don't tell you whether your agent hears its callers.
What the benchmark actually measures
Worth being precise about this, because a lot of teams read the top-line number and stop reading. The Pipecat benchmark plays pre-recorded PCM audio through a Pipecat pipeline in 20 ms chunks at 16 kHz, exactly as a live microphone would deliver it, uses Silero VAD to mark start/stop speaking, and measures from the VAD going low to the final transcript segment arriving. Semantic WER is scored by asking Claude to judge whether the transcription errors would matter to a downstream LLM ("gonna" vs "going to" doesn't; "Corel" vs "Coral" does). The dataset is 1,000 real agent-style utterances from smart-turn-data-v3.1, most of them a few seconds long.
That's a clean, well-designed component benchmark. The word to hold onto is component. It's grading one box in the pipeline. It's grading it on isolated utterances, on clean 16 kHz audio, in a synthetic transport, in one language, with no other speaker on the line, and it's scoring an LLM-judged notion of "meaning" that isn't specific to what your agent is actually doing with the transcript.
Five things a semantic WER leaderboard won't tell you
Here's the list I keep going back to when a team asks me why their agent still misheard the caller after switching to whichever provider is currently at the top of the chart.
1. Your codec. The Pipecat benchmark runs 16 kHz PCM. Your production runs 8 kHz μ-law over G.711 because that's what telephony hands you. Half the frequency spectrum, half the bit depth, and a nonlinear companding curve applied by the carrier. AssemblyAI's own guide for phone-based agents recommends native 8 kHz μ-law support and strong accuracy on alphanumerics as separate criteria from raw benchmark accuracy for a reason. Models trained more on clean conversational audio than telephony show it the moment you put them on a Twilio call.
2. Your entities. Semantic WER penalizes only errors that a general LLM judge thinks change meaning. But your agent isn't a general LLM. It's writing a confirmation code into a CRM, or a drug name into an EHR, or a routing number into a payments API. "Corel" vs "Coral" changes meaning for your business even when the judge says it doesn't, and the missed entity rate on proper nouns, alphanumerics, emails, and addresses is a separate metric for a reason. AssemblyAI's own Universal-3.5 Pro Realtime posts a 15.31% entity error rate on real agent conversations, on a model that's leading the semantic WER frontier. The top-line number does not predict the entity number.
3. Your callers. smart-turn-data-v3.1 is a fixed dataset. Your callers are not. If your agent takes inbound calls from senior patients in Alabama, or from tradespeople yelling over a running compressor, or from callers who switch mid-sentence between English and Bahasa (which Seavoice built a whole product around), none of that is in the benchmark. And background noise is doubly nasty: audio enhancement solutions can improve perceptual audio quality while making WER worse, so the fix that sounds better isn't guaranteed to be the fix that transcribes better.
4. Overlap. The benchmark plays one speaker into a clean pipe. Your production is duplex. The caller talks over the agent's TTS, the STT sees a mix, the VAD flaps, and the "final transcript" the benchmark is measuring is a very different beast from the transcript your context aggregator receives when the caller barges in at the 400 ms mark. Barge-in requires semantic VAD on the STT side and instant TTS interruption on the playback side, and none of that is being graded on a synthetic transport.
5. The rest of the pipeline. ASR errors compound. A 1.2% semantic WER on a 5-second clip becomes something else once your LLM writes the next question on top of it, your TTS says the mis-heard name back to the caller, and the caller either corrects it (and now you're re-doing the previous turn) or hangs up. There's a paper on LLM-based conversational agents where WER hit 19.7% on one non-native speaker and the downstream answer accuracy was still high, and inverse cases too. Transcription accuracy does not linearly translate to conversation accuracy, in either direction.
Testing ASR the way your agent uses it
The uncomfortable version of the argument is: even a perfect component benchmark, run correctly, is not your acceptance test. The comfortable version is that it's still useful. Semantic WER is a floor: if your STT sits below the Pareto frontier on a public benchmark, that's a real signal, and you should switch. But everything above the floor has to be measured on your agent, on your traffic, in the format the phone carries.
Three things that work:
- Simulate over the phone, not over a socket. Personas dialing your agent's actual number (or your agent dialing them), with the accents and pace and emotion and background noise of your real callers, running on a schedule and triggerable from CI so a bad model swap fails the build. Not text loopback, not synthetic transport into a Pipecat pipeline: real telephony carrying real μ-law from a real endpoint.
- Score the audio, not just the transcript. Missed entity rate on the terms your agent captures, mispronunciation and prosody on the terms your agent speaks, interruption and dead-air handling on the moments your agent is expected to yield. Every one of those breaks in ways the transcript alone cannot see. This is the point HackerNoon's roundup makes bluntly: read the transcript alone and the call looks fine, listen to the call and it's a disaster.
- Replay production failures. When a real caller says "Ontario Route 27" and your production call log shows the agent booked "Ontario Room 27", that call is more valuable as a regression test than any synthetic scenario you could write. Capture it, replay it against every new STT version, every new LLM version, every prompt change.
That set of three is what an "ASR test" actually looks like for a voice agent. Not one number, not one dataset, and not one component in isolation.
Where Roark fits
We built Roark because the semantic WER leaderboard is where the interesting work stops, and where the interesting work for a production voice agent starts. Simulation testing dials your agent over real PSTN and WebRTC, in 45 languages and accents, with personas that vary voice, speech pace, emotional register, and background-noise environment. Both inbound and outbound, and simulations can be triggered over HTTP so a scheduled run or a CI job gates a deploy. That's how you get ASR grading that reflects the audio your callers actually deliver.
Every production call gets scored against 64+ built-in audio-native metrics, plus whatever custom metrics you define for the entities your business actually cares about. The scoring is audio-native, not transcript-only: Roark grades pronunciation, emotion, vocal stress, pace, pauses, and interruptions. When a metric fails, an issue is filed automatically. And when a real production call fails in an interesting way, you can capture and replay it against updated agent logic, turning that specific misheard entity or that specific botched barge-in into a repeatable regression test. There are one-click integrations for Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs, so wiring this into an existing stack is a config change, not a rewrite.
The leaderboards are useful. Pipecat, AssemblyAI, Soniox, and Speechmatics have made voice-agent STT evaluation genuinely better this year, and I'd rather live in a world where semantic WER is the default than one where raw WER still is. Just don't confuse "our STT provider is on the Pareto frontier" with "our agent hears its callers." Those are different tests, and only the second one keeps you off the postmortem.
