Real scale is finally showing up. In a two-week public campaign this month, Japanese home-appliance retailer Yamada Denki's voice agent held sales conversations with about 30,000 shoppers, with 92% of post-conversation survey responses positive. In healthcare, Assort Health closed a $120M Series C at a $1.2B valuation, and Harvey led the month with a $200M Series C. The pattern is the same across verticals: voice agents that lived on a few dozen pilot calls a week are about to face hundreds of them at once.
Launch QA has not caught up. Most acceptance plans I see grade one call at a time. One scripted persona, one quiet room, one caller. The agent passes. Then the campaign fires, or a service outage triggers a notification blast, or Monday morning starts, and the same agent starts leaving dead air where its own response should be. Concurrency is the launch metric almost nobody tests for, and it is the one that decides whether a pilot survives contact with a real week.
The demo-to-peak gap is measurable
The demo-to-production drop is not a soft claim. One industry write-up puts it plainly: under production load of 100+ concurrent calls, latency typically degrades 40 to 120% from demo conditions, and roughly 60% of deployments that pass demo evaluation fail within 90 days of launch. Whatever your view on the exact numbers, the direction is uncontroversial. The Retell team says the same thing from the platform side: scaling issues surface long before traffic hits extreme levels, and once dozens or hundreds of concurrent conversations are running, weaknesses in concurrency handling, audio streaming, and orchestration become visible.
The real question is what "under load" hides that single-caller QA cannot see.
What actually breaks when calls stack up
Some of these show up in every scaling post-mortem I have read:
- First-token latency stretches, blowing the sub-800ms p95 that separates natural conversation from callers interrupting the agent.
- ASR errors climb, because concurrent calls typically also mean noisier calls, and a single-persona QA suite never covered the acoustic distribution your real callers bring.
- Downstream tool APIs rate-limit, and the agent goes silent while it waits for a CRM lookup or a scheduler query that timed out.
- Session state leaks between concurrent calls. As one production post-mortem put it, under load or a session-management edge case, an agent references information from a previous caller's session. This one is rare in testing precisely because nobody runs the system at the volume and concurrency of real production.
- Escalation flow queues. Warm transfers and human handoffs share the same threadpool as the agent, and when volume triples, escalation stops firing cleanly.
None of these are model-quality problems. All of them are surfaced only when the load looks like production.
Why single-caller QA misses it
The way most teams run acceptance testing has three structural blind spots.
The suite runs sequentially. Twenty scenarios executed one after another cannot reveal a concurrency bug, because there is never more than one call in flight. A cheap trick is to run the same twenty in parallel and see what changes. Usually a lot does.
The persona is uniform. Voice AI teams often lock QA to a single synthetic voice reading a script cleanly. Real peak traffic is a distribution: accents, ages, background environments, emotional registers. Latency budgets that pass on the QA voice do not automatically pass on a distressed caller in a car.
The metrics are averaged. The mean tells you almost nothing. Callers hang up on p95 outliers, not means. If you only report an average latency, a bimodal distribution (most calls fine, 8% catastrophic) looks acceptable. It is not.
A PM playbook for concurrency testing
The plan does not have to be elaborate. It has to be honest.
1. Pick the peak, not the average. Model 2x your expected peak concurrency, whatever your forecasting method. If you expect 80 concurrent calls at 10am Monday, test at 160. Peaks are where you find the failure modes; the average is where you already know it works.
2. Build a persona mix that mirrors the caller distribution. Not one voice repeated 160 times. A weighted set that reflects the accents, languages, speech pace, emotional registers, and background environments in your real traffic. If 12% of your callers speak Spanish and 8% call from a car, your load test should look like that.
3. Fire in parallel over real telephony. A text-loopback harness cannot expose jitter, packet loss, TTS chunking under contention, or endpointing under real audio. If the production channel is PSTN, the load test is PSTN.
4. Score every simulated call individually. Aggregates lie. You want the full distribution of first-token latency, dead-air incidents, ASR error rate, task-completion, escalation-fired rate, cost per successful call, per persona bucket.
5. Rerun on every stack change. Model swap, TTS switch, orchestration change, prompt rewrite: rerun the peak suite. Latency benchmarks are only meaningful when conditions are reproduced; yours are the ones that matter, and they are only stable if you rerun them.
What to measure at load
A short list, ranked by how often they catch real launch blockers:
| Metric | Why it matters at load |
|---|---|
| p95 first-token latency | Averages hide the outliers callers hang up on |
| Dead-air incidents per 100 calls | Silence is the failure mode callers notice first |
| Interruption / barge-in handling | Endpointing degrades fast under contention |
| ASR word error rate by persona bucket | Reveals accent and noise blind spots in QA |
| Task-completion rate at peak | The only metric your business actually cares about |
| Escalation fire rate | Should be stable at load, not spike or vanish |
| Cost per successful call | LLM and TTS costs accrue by the second under load, and peak is where unit economics get real |
Any one of these can be a launch blocker on its own. All of them can be captured from a well-instrumented concurrency test.
Simulation testing is how you do this without a real Monday
You can, in theory, find all of this out by launching and watching what happens. It is the most expensive way to run a load test ever devised, and the invoice arrives as churned callers.
The alternative is to run the peak against the agent before launch, from personas and scenarios you control, over real telephony, and grade every call. This is what Roark is built for. Simulation runs dial your agent over real PSTN and WebRTC in parallel, from persona definitions that carry accent, language, pace, emotional register, and background environment. Runs are packaged as scenarios and run plans and can be scheduled or triggered from CI over HTTP, so the peak suite gates the deploy. Every simulated call, and every production call after launch, is scored against the built-in audio-native metric suite (64+ metrics, plus any custom ones you define) covering pronunciation, emotion, vocal stress, pace, pauses, and interruptions. What breaks is filed as an issue automatically. One-click integrations cover Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs; if your stack is custom, the SDKs push calls in.
The point is not just running the test. It is turning peak-load failures into a durable regression asset.
Turn production peaks into replayable regressions
The most underused testing surface most teams have is their own production call history. The best argument I have seen for it says: replay real production calls against new agent versions, especially the ones that previously failed or escalated. Synthetic scenarios cannot cover everything real callers do at real conditions.
Roark's production call replay captures real calls and replays them against updated agent logic, so a failure at last Tuesday's 11am peak becomes a scenario you run before Wednesday's model swap. That is what makes load testing compound: each peak you survive contributes calls to the corpus that grades the next launch.
Launch-readiness checklist
Before you ship the change, ask:
- Have you run the suite at 2x expected peak concurrency, over real telephony?
- Do you know p95 first-token latency and dead-air rate at that load, not just at n=1?
- Does your persona mix match the caller distribution, including accents, noise, and non-cooperative callers?
- Does the escalation path still fire under load, and does the receiving human get full context?
- Have you captured last peak's failures as replayable scenarios in your regression suite?
- Does the peak suite rerun on every stack change (model, TTS, orchestrator, prompt), and does the CI job actually gate the deploy?
None of this is exotic. It is the difference between the launch that quietly works and the launch that produces a post-mortem.
The short version
Voice AI is moving from pilots to volume. The teams that keep passing acceptance while their production quality collapses are the ones still grading one caller at a time. Load is not a nice-to-have piece of QA. On a phone line, it is the launch metric.
If you want to see what scored concurrency runs look like against your stack, Roark will dial your agent in parallel from real personas today.
