On July 6, OpenAI shipped gpt-realtime-2.1 and gpt-realtime-2.1-mini, cutting p95 latency across its Realtime voice models by at least 25% through improved caching. Twenty-seven days later, Article 50 of the EU AI Act became enforceable, and every voice agent taking EU calls now has to disclose that it is AI at the first interaction. Neither event required you to push code. Both change what your live agent does on the phone.
That is the shape of the problem this post is about. The model beneath a production voice agent is not a fixed asset. It moves on the provider's schedule, not yours, and every move is a regression event whether you treat it as one or not. This is a technical playbook for building a harness that catches those regressions before your callers do.
The three things that actually change
When a voice model updates under a live agent, three axes shift, and they interact.
Timing. The clearest example is the July release: a 25% p95 latency cut sounds like a pure win, but every agent whose barge-in behavior was tuned against the old timing now has different endpointing dynamics. OpenAI attributes the reduction to improved caching, which means responses can start faster on some turns and not others, depending on prompt reuse. Turn-taking and interruption code that assumed a specific response distribution starts firing in new places.
Behavior. New model weights change what the agent decides to say. GPT-Realtime-2 already introduced adjustable reasoning-effort levels and a 128k context window that was four times larger than its predecessor, and 2.1 layered reasoning into the mini tier at the same cost. That is enough to reshape tool-call ordering, retry logic, and how the agent handles ambiguous requests, even when the prompt is untouched.
Voice. Speech-to-speech models emit audio directly, so a model swap can subtly change pacing, prosody, and even pronunciation of proper nouns and policy identifiers. A regression here is invisible to a transcript diff and often invisible to a human ear until a caller complains they were interrupted or confused.
None of this is hypothetical. This is what the production Realtime API is now shipping into stacks that were validated against last quarter's model.
Why standard regression testing misses model swaps
Most teams inherit their voice-agent testing from LLM-app patterns: a suite of scripted inputs, an eval that compares outputs to reference answers, maybe an LLM-as-judge for open-ended turns. That harness has three blind spots that model swaps exploit directly.
- It runs on text, not audio. The transport, the codec, the ASR, the TTS, and the network all matter for what a caller actually hears. A text loopback test cannot see that the new TTS mispronounces your product's name, or that endpointing now cuts the caller off 200ms earlier.
- It scores content, not conduct. Whether the agent said the right thing is one axis. Whether it interrupted, hurried, hesitated, or sounded stressed is another. Model updates hit both, and the second one is what callers actually notice.
- It uses synthetic-only inputs. Scripted personas cover the shape of a conversation. They do not cover the specific accents, background noise, and phrasing your real callers use. A regression that only shows up on a Glasgow accent at 2am on a noisy line is not going to surface in your fixture set.
The fix is not to write more transcript assertions. It is to build a harness that runs real audio over real telephony and scores what the caller hears.
The five-stage harness
The harness has five stages, and each one has to exist for the whole thing to be useful.
1. Capture and tag production calls
Every production call gets recorded and stored with metadata: which flow it entered, which tools were called, which model version served it, whether it hit an escalation, whether the caller sounded frustrated. This is table stakes for observability, but for regression the tagging is what matters. You will not go back and hand-label 40,000 calls when a new model drops.
2. Curate the replay set
The replay set is not "all of production." It is the smallest collection of real calls that, together, exercise every path you care about: the happy path in each supported language, the interruption paths, the disambiguation paths, the escalations, the calls that failed last time. Voice agents fail in clusters, so aim for coverage of the failure modes you have already seen, not exhaustive coverage of every branch.
Rotate the set. Calls that stop finding regressions get archived. New failure classes get promoted in.
3. Simulate the replay set against the candidate model
This is where most homegrown harnesses fall over. Replaying a call as a text transcript is not the same as replaying it as a phone call. To catch timing, prosody, and turn-taking regressions, the replay has to happen over the same transport your production agent uses, with a caller voice that has the same pacing and background characteristics as the original. That means real telephony (PSTN or WebRTC), synthetic caller audio driven by personas that mirror the tagged production traffic, and an agent endpoint pointed at the candidate model.
4. Score with audio-native metrics
Transcript-based scoring rules out the entire class of regressions that model swaps most reliably introduce. You need metrics that read the sound of the call: pace, pause distribution, interruption detection, dead-air spans, pronunciation accuracy on named entities, emotion and vocal stress in both directions. Content metrics still matter for did-it-book-the-appointment questions, but they run alongside audio-native ones, not instead of them.
5. Gate CI on the diff
The output of the harness is a diff against the last known-good run, per scenario, per metric. If any metric moves outside its threshold, the pipeline blocks. This is where a provider-side model swap gets caught: your nightly run against the current production endpoint drifts, and the alert fires before you have shifted traffic.
Two implementation notes for teams wiring this up. First, the harness must be triggerable by HTTP, not just cron, so it can gate CI on your side and be re-run on demand when a provider announces an update. Second, the threshold values should live in code, in your repo, alongside the scenario definitions. Threshold drift is its own regression.
Where Roark fits
Everything above describes an approach. Building it from scratch is a six-month project with a full-time owner. Roark is the harness as a product.
Roark's simulation testing dials your agent over real PSTN and WebRTC using personas that model caller voices, languages, accents, pace, emotional register, and background-noise environments across 45 languages. Simulations are built from scenarios, run plans, and schedules, so a regression suite reruns automatically or on an HTTP trigger from your CI. Every call, both simulated and production, is scored against 64+ built-in audio-native metrics plus any custom ones you define, and what breaks is filed as an issue with the call attached.
The production-replay capability is the piece that maps most directly to model swaps. Roark captures real production calls and replays them against updated agent logic, turning the calls that failed against your last model into repeatable regression tests against the new one. One-click integrations exist for Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs, and calls can be ingested via the Node or Python SDK with client.call.create and a recording URL. Full details are in the docs.
Article 50 is a good stress test
The EU AI Act's transparency chapter is a useful worked example because it is precise, testable, and enforced. From 2 August 2026, an AI phone agent operating in the EU must disclose that it is a machine and on whose behalf it is calling, spoken, in plain language, at the latest at the first interaction. Buried privacy-policy disclosures and vague labels do not meet the standard. Breaches sit in the middle sanctions tier of the Act.
That is a metric. Every EU-bound call should be scorable on:
- Was the disclosure spoken in the greeting turn?
- Did it name the deploying organization?
- Was it in the language of the call?
- Did it land before the caller's first substantive utterance?
Wire those checks into your metric suite and every model swap gets stress-tested on a regulation that carries real fines. If a latency change or a prompt-caching quirk causes the disclosure line to be cut short on 4% of German-language calls, you want that in a CI failure, not in a supervisory-authority letter.
What good looks like
A team running this well has a few properties in common:
- Nightly simulation runs against the production endpoint. Not just pre-deploy. Provider-side changes are caught by the nightly diff, not by the pre-merge gate.
- The replay set is a rotating asset. Someone owns it. New failure classes get promoted in within a week of first observation.
- Every metric has a threshold in code and a plain-English rationale. "Interruption rate under 3% because callers rate anything higher as rude in our post-call survey," not a magic number.
- Model releases are treated as deploys. When OpenAI's changelog announces a Realtime model bump, the on-call rotation runs the suite before shifting traffic, and rolls back on a metric regression the same way they would a bad prompt.
- The suite runs on a schedule the reviewer trusts. If the harness only runs when someone remembers to trigger it, it will not catch the 3am cache-warming behavior change.
The next model is already in the queue
Between May and July 2026, the Realtime lineup went from gpt-realtime to gpt-realtime-2 to gpt-realtime-2.1, with the mini tier picking up reasoning at the same price. LiveKit Agents 1.5 shipped adaptive turn detection and native MCP tool support, Pipecat reached 1.0 in April, and the whole ecosystem is on a monthly release cadence.
The next model that changes your agent's behavior is already in someone's dev branch. The question is not whether it will land under you. The question is whether the harness that scores it is running before or after your callers notice.
