LiveKit Agents 1.8.11 landed last week with two beta additions that quietly change what a "voice agent" can be: OpenAI's gpt-realtime-translate and Google's gemini-3.5-live-translate-preview, both wired through a new TranslationModel interface. These are not LLMs you talk to. They are interpreters you bridge between two humans on a call, or between a caller and the agent beneath, in real time.
If your test suite was built for a one-speaker, one-language voice agent, it does not cover any of the failure modes an interpreter introduces. This post lays out what breaks, how to simulate it before launch, and what to score on every live translated call.
What an interpreter model is, and is not
Google and OpenAI are explicit about the positioning. Gemini 3.5 Live Translate functions as a real-time translation pipeline without function calling, search grounding, tools, or system instructions. OpenAI similarly describes gpt-realtime-translate as an interpreter rather than a voice assistant, aimed at empowering humans to be multilingual rather than building AI voice agents. If you are building voice agents, OpenAI tells you to use gpt-realtime-2 instead.
That design choice has consequences. The interpreter model does not take a system prompt, does not accept tools, and does not steer mid-stream. It uses dynamic voice adaptation: translated speech follows the source speaker's tone and pitch, so the voice changes as speakers change. It carries no in-line speaker attribution in the synthesized audio, and once audio is emitted, you cannot retract it the way you can rewrite displayed text.
And the quality tradeoff is the one most teams miss on first read: the model is optimized for fluency and low latency, not verbatim accuracy. It is for natural conversation flow, not precision documentation.
Where interpreter mode shows up
Three patterns are in production now.
Bridged calls. Caller speaks language A, the human or AI agent on the other side speaks language B. Two translation sessions run in parallel. The OpenAI cookbook says this explicitly: for a two-person call, create two translation sessions, A-to-B and B-to-A.
Multilingual agents with an interpreter layer. The primary agent runs on a conversational model (gpt-realtime-2, Gemini Live, or similar), with a translation sidecar for callers who drop into a language the main agent does not cover well.
Broadcast and one-to-many. A single speaker, N listener-specific translations. Meetings, telehealth classes, education. LiveKit has reference implementations for both Gemini and gpt-realtime-translate that cover this pattern.
The bridged-call pattern is the one causing the most testing debt. It is a voice agent product built from three independent streaming services (ASR, translator, TTS) glued to two audio channels, and nothing in your existing suite knows how to score it.
The failure modes that do not exist in a normal voice agent
Latency compounds across both directions
A normal voice agent's time-to-first-audio is one number. An interpreter has two, one per direction, plus the gap each caller waits between finishing a sentence and hearing a translated reply from the other side. Even with gpt-realtime-translate's typical 300ms to 800ms end-to-end latency on a stable connection, a bridged call accumulates that twice per exchange.
Chained pipelines (Whisper, then GPT, then TTS) add 2 to 4 seconds per direction, which pushes a bridged call well past the point where turn-taking breaks down. If you are moving from a chained pipeline to a direct interpreter model, the latency win needs to be measured, not asserted.
Hallucinated translations and silent omissions
Fluency-optimization is the honest headline on these models. For casual conversation that is fine. For a patient reading back a prescription dose, or a caller spelling a legal name over the phone, "fluent approximation" is a defect, not a feature.
You cannot test for this by diffing transcripts. Source and target are in different languages, and the model's output is audio, not text. You need back-translation on the delivered audio, or glossary-aware scoring that checks specific tokens survived the trip.
Speaker attribution goes blurry
Dynamic voice adaptation is a lovely feature until you have three or more speakers. The translated output mimics whichever source speaker was last heard, and there is no in-line attribution in the audio itself. Listeners hear three different voices on their side (original A, original B, translated A or translated B) with no cue for which is which. If your product surfaces subtitles, test the attribution those subtitles carry. If it does not, test what the agent does when it guesses wrong about which side it is translating.
Code-switching mid-utterance
Real callers drop proper nouns, numbers, and courtesies in another language. A patient speaks their name in English during an otherwise Spanish call. gpt-realtime-translate claims to support language switching mid-stream, but behavior in telephony-grade audio is uneven. You want this in your test suite with fixtures, not caught in production.
Domain terms get smoothed
Interpreters optimized for fluency rewrite phrases that do not translate cleanly. "Prior auth." "Copay." "Deductible." "Parts markup." These are tokens your human agent treats as fixed, and the translation model will cheerfully paraphrase them into something that sounds right and means something adjacent. A test harness for a healthcare or home-services deployment needs a glossary check on every translated call.
No instructions means no in-session mitigations
Because the interpreter will not accept a system prompt or tools, you cannot tell it "translate copay as copago, never coparticipación." Your knobs are: which model, which pair of languages, and whatever post-processing you layer on top. All your constraints live outside the translation session, which means they have to be tested against it, not configured into it.
Barge-in gets strange
In a normal voice agent, barge-in is: caller interrupts, agent cancels TTS, agent listens. In a bridged interpreter, two humans interrupt each other, and both translation streams have to cancel the right partial audio on the right channel. Overlap handling across two full-duplex translation sessions is a real engineering problem, and it deserves a test suite of its own.
What to simulate before launch
The non-negotiable scenarios for an interpreter-mode agent:
- Latency ladder. Same conversation under clean WebRTC, bonded cellular, 2% packet loss, 150ms jitter. Measure both directions independently, plus the per-turn gap each caller experiences.
- Glossary adherence. Scripted calls that force domain terms through each language pair. Score on whether the term survives verbatim.
- Numbers, spellings, and names. Digits spoken as numbers vs digits as names ("my policy is A, K, four, two, seven"). Interpreter models mangle both frequently enough that this is its own scenario class.
- Code-switching. Primary language with injected foreign-language names or quantities. Score on whether the translation preserves them.
- Overlap and barge-in. Scripted interruptions from both sides, both before and after translated audio has started playing on the opposite channel.
- Noise and accent drift. The same persona under café noise, car background, and strong regional accent. Fluency-optimized models degrade quietly rather than refusing, which is worse than failing loudly.
- Long silence and dead air. Interpreter models sometimes refuse to emit until enough source audio accumulates. Measure that threshold per language pair.
- Session cold start. First-word latency from a fresh session, because translation sessions are typically created per-call.
These have to be real phone calls. Text-level eval does not catch a single one of the above, because every failure mode is audio-shaped: timing, pronunciation, interruption, speaker tone, background. A simulation suite that dials the agent over PSTN or WebRTC and scores the resulting audio is the only honest way to cover them.
What to score on every live call
Simulation stops being enough the moment you go live. The call that delivers a wrong dosage through translation is not in the test suite; it is in production tomorrow. Interpreter-mode deployments need runtime scoring on the same audio-native dimensions you tested for:
- Translation fidelity (back-translation delta on scripted terms)
- Per-direction time-to-first-translated-audio
- Per-turn gap distribution
- Glossary adherence (term presence in output transcript)
- Overlap-cut cleanness (did the right channel cancel on barge-in)
- Pronunciation score on named entities
- Source-silence vs translated-silence ratio (interpreter stalls)
Every one of these is derivable from the call audio plus a minimal transcript. None of them require the translation session to be instrumented. You need them running on every live call, filing an issue the moment a metric trips, and routed to the engineer who owns that pair of languages.
Dual-channel STT helps here. Soniox's stt-rt-v6 release lets you send the agent's own text back over the same WebSocket so the recognizer uses it as context and removes the agent's voice from the transcript. That makes per-channel scoring much cleaner on bridged calls where both sides are talking into the same audio mixdown.
Where Roark fits
Roark is a simulation-testing, observability, and reporting platform purpose-built for voice agents, and interpreter-mode agents are a case it fits cleanly.
The simulation side dials your agent over real telephony. Personas carry languages, accents, pace, emotional register, and background noise, which is exactly the matrix you need for scenarios 2 through 7 above. Roark simulates and scores across 45 languages and accents, so you can build the pair-matrix you actually ship: Spanish to English, Tagalog to English, Portuguese to Spanish.
The scoring side runs on every production call. Roark's audio-native models score pronunciation, pace, interruptions, and emotion on the delivered audio, which is the layer where interpreter failures live. 64+ built-in metrics ship with the platform, and custom metrics cover the domain-specific checks (glossary adherence, per-direction latency) that a healthcare or insurance deployment needs. When a call fails a metric, Roark files the issue automatically.
Production call replay closes the loop. When a real translated call goes wrong, you replay that exact audio against the new model, the new prompt around the model, or the new TTS voice, and see whether the fix held. Interpreter models update behind the API, and without replay, every provider refresh is a silent deploy against a critical path.
One-click integrations are in place for LiveKit, Pipecat, Vapi, Retell, Bland, and ElevenLabs. For a LiveKit Agents stack running the new beta TranslationModel, the wiring takes minutes. The docs walk through SDK ingest and OTEL traces.
A short launch checklist
Before you ship an interpreter-mode voice agent:
- Every active language pair has a simulation suite that exercises latency, glossary, code-switching, and barge-in.
- Pre-launch simulations run on real telephony, not text eval, and are scored on audio.
- Runtime metrics cover both directions independently, with per-direction alerts.
- Production call replay is wired, so provider updates can be re-tested against real failure cases.
- Domain glossaries are codified and scored, not left to the translator's fluency bias.
- A known-good baseline is captured on day one so you can detect drift when the model refreshes.
Interpreter models are a genuinely new primitive for voice agent builders. They collapse a three-stage pipeline into one session, cut latency to the point where cross-language calls feel conversational, and arrive with a set of failure modes that nothing in your existing test suite covers. Ship them with a simulation suite that speaks the languages you serve, and an observability layer that scores the audio they actually produce.
