Your transcriber doesn't run alone. Neither does your voice. So why do we benchmark them that way?
Today we're making Voice AI Benchmarks public. It ranks over 650 complete voice AI stacks on nearly 14,000 scored live phone calls, across 12 providers: Amazon, Anthropic, AssemblyAI, Cartesia, Deepgram, ElevenLabs, Google, OpenAI, Rime, Speechmatics, Ultravox and xAI.
Why component leaderboards miss the point
Most voice AI leaderboards score one piece at a time. There's a speech-to-text board with word error rates on clean clips, and a text-to-speech board with listener preference scores. They're useful, but they answer a narrower question than the one teams actually face.
Callers never hear a model in isolation. They hear a transcriber, a language model and a voice working together over a phone line, with real latency, real interruptions and real tool calls. A model that tops its own chart can still sink the stack it lands in.
We see this constantly at Roark. We build observability and simulation for voice and chat agents, so we spend our days watching how agents behave on real calls. The question teams ask us isn't "which STT is best?" It's "which combination should I ship for my calls?" We wanted a public answer to that question that we could back with data.
What we built
Voice AI Benchmarks runs full pipelines end to end on live phone calls:
- Cascades: an STT model, an LLM and a TTS model chained together.
- Speech-to-speech models: a single model that listens, reasons and speaks, including releases from OpenAI, xAI, Google, ElevenLabs, Amazon and Ultravox.
Both types run the same agent on the same calls, so you can compare them directly. Each model is ranked on every stack it runs in, so its score reflects how it does alongside the components around it, not just on its own.
How it works
Every stack handles four real jobs: a healthcare appointment reschedule, an urgent home repair, an auto insurance claim and an internet outage. Each job is played by the same five callers, ranging from a quiet US line to a fast talker interrupting from a construction site.
We score each call on more than how it sounded:
- Task completion: did the agent actually get the job done?
- Tool use: did it call the right tools with the right arguments?
- Time to first word and response time: how long callers wait.
- Barge-in handling: what happens when the caller interrupts.
- Cost per task: what a completed task actually costs.
You can rank on overall quality, or on a value score that adjusts for price.
We also try to be honest about uncertainty. Every score carries a margin of error, and a stack's rank is the best place it could hold within that margin. That means ties are real and shown as ties: on the overall performance board, 85 stacks share first place. If two stacks are statistically indistinguishable, we'd rather tell you that than invent a winner.
What we found
A few things stood out in the latest sweep.
The same model can land almost anywhere. AssemblyAI's Universal-3.5 Pro transcriber appears in stacks ranked anywhere from #1 to #575, depending on what it's paired with. This is the clearest case for benchmarking stacks: a component's rank on its own tells you surprisingly little about how your agent will perform.
The language model moves results the most. Swapping the LLM changes task completion by about 7 points, against 3 for the speech engine and 2 for the transcriber. Averaged across every cascade it appears in, OpenAI GPT-6.1 Sol leads the overall performance board (61.0), followed closely by Anthropic Claude Sonnet 5.5 (58.8) and Google Gemini 3.8 Flash (58.7).
At the top, completion is nearly solved. Speed and cost decide. 430 of 642 cascades complete the task in at least 90% of calls, and the top 10 sit within one point of each other. Meanwhile median response time runs from 2.0 s to 6.2 s on the same agent and the same caller, a spread that comes entirely from the models behind it.
Cascades vs speech-to-speech is a real tradeoff. Comparing the three strongest of each, cascades lead on task completion (94% vs 85%), time to first word, barge-in handling and cost per task ($0.15 vs $0.24). Speech-to-speech leads clearly on response time (2.1 s vs 3.8 s). Neither wins outright, so the right choice depends on what your calls need.
The highest-ranked stack on the value board right now is AssemblyAI Universal-3.5 Pro + OpenAI GPT-6.1 Sol + Rime Coda, but with the margin of error, several others share that spot. That's the point: the board shows you the shortlist, and your own calls decide.
How to use it
- Choosing a stack: filter by industry and sort by the metric that matters most for your calls, whether that's task completion, latency or cost.
- Comparing two options: use the side by side comparison to see exactly where two stacks differ.
- Evaluating a model swap: check how a new model performs across every stack it appears in, not just its best one.
Benchmarks are a starting point, not a verdict. Your prompts, tools and callers are different from ours. The best way to know how a stack performs for you is to test it on your own scenarios before callers hear it, and keep watching it once it's live. That's what we built Roark for.
Dig in
The benchmarks are public and free to explore: benchmarks.roark.ai
We'll keep running new sweeps as models ship. If you'd like to see how your own agent stacks up on your real scenarios, book a demo and we'll walk you through it.
And we'd love to hear from you: which combination would you bet tops the board?
