Testing TTS pronunciation in voice AI agents

Your voice agent knows exactly what to say. It just says it wrong. The caller hears "adalimumab" as "add-a-LEE-mumab," or "Nginx" as "en-jinks," or your own brand name as something the marketing team quietly winces at. The transcript is perfect. The eval is green. The caller still hung up.

TTS pronunciation is the QA gap most voice teams underinvest in until a customer complains, and it is the category of failure that text-only regression tests are structurally blind to. A model swap, a voice change, or a dictionary version bump can silently ship a mispronunciation into every call, and nothing in the transcript will tell you. This post is a technical playbook for testing pronunciation at the layer it actually lives in: the audio.

Why neural TTS mispronounces the words that matter most

The words that matter most in a conversation are exactly the words TTS is worst at: proper nouns, brand names, drug names, addresses, product SKUs, and acronyms. These are almost by definition out-of-vocabulary. As the Onepin writeup on brand-name pronunciation puts it, TTS models are trained on general-language corpora, so their phoneme mappings reflect common words. Brand names, product names, and proper nouns rarely appear in training data, and the model has no domain-specific context to override its default guess.

Older concatenative TTS engines had an explicit lexicon you could patch. Modern neural TTS makes this worse, not better. Gaudiy's engineering post on custom pronunciation notes that end-to-end neural models learn pronunciation implicitly, which makes it harder to override any specific word, and their output is context-dependent in ways you cannot predict from the text alone.

Concretely, the failure categories that show up in production voice agents:

• Proper nouns: personal names, company names, street names, cities. "Leicester," "Xiaomi," and "Hermès" fail predictably in English models.
• Acronyms and initialisms: some spoken as words (NASA), some as letters (FBI), some genuinely ambiguous (SQL, GIF, API). "Dr." should become "Doctor." "St." could be "Street" or "Saint" depending on context, and the model applies a single default that ignores context entirely, per the same Onepin analysis.
• Medical, legal, and technical jargon: drug names, procedure names, statute names, tool names. Getting "adalimumab" or "kubectl" wrong is a credibility hit in exactly the verticals where credibility is the product.
• Cross-lingual proper nouns: ElevenLabs Magazine's guide notes that proper nouns from non-English languages, including names of people, cities, companies, and places, are consistently among the hardest for AI models trained primarily on English text. French, German, Japanese, Chinese, and Arabic names have phonetic patterns that English orthography does not represent, and models default to English pronunciation rules.
• Alphanumeric strings and numbers: confirmation codes, order numbers, account IDs. Cartesia called this out as a headline improvement in Sonic 3.5, which added more natural pacing, correct alphanumeric read-out for confirmation codes and phone numbers, and context-aware heteronym pronunciation when it went GA in May 2026.

The known fix, and where it silently breaks

The industry answer to all of this is custom pronunciation dictionaries. ElevenLabs, Cartesia, and every platform that wraps them have converged on the same shape: alias rules that swap a word for another spelling, and phoneme rules written in IPA or CMU Arpabet.

Vapi exposes them as first-class configuration on the assistant. That is genuinely good tooling. The catch is that the moment you look at it, you see the acceptance surface you now own:

• ElevenLabs phoneme rules only work with eleven_turbo_v2 and eleven_flash_v2, and Cartesia pronunciation dictionaries only work on the sonic-3 or newer model.
• Alias rules apply pre-model as a text substitution. Phoneme rules apply per-language. Rule order matters, and matching is case-sensitive.
• Voice markup is provider-specific. Vapi's own docs warn that xAI uses speech tags while Cartesia uses SSML-like tags, and Vapi does not translate markup between providers, so a tag supported by one provider might be spoken aloud verbatim by another instead of being applied.

Every one of those constraints is a regression waiting to happen. Swap your voice from an ElevenLabs Flash to a Cartesia Sonic and every phoneme rule silently stops applying. Roll the voice from sonic-3-2026-01-12 to a newer snapshot and the pronunciations you validated last quarter may drift. Add a new language and every alias you wrote in English does nothing when the phrase comes out in Spanish.

This is precisely the pattern the Cartesia Sonic-3.6 writeup warns about: language availability does not guarantee equally strong accents, names, code-switching, or domain pronunciation in every locale, and because model aliases and behavior evolve, teams should regression-test updates and pin versions when stability matters.

The dictionary is not the test. The test is what proves the dictionary still works after every change.

Why transcript-level evals miss all of it

Nearly every voice-agent test harness in the wild scores the transcript. That is fine for prompt regressions, tool-call correctness, and instruction-following. It is useless for pronunciation, because pronunciation is not in the transcript by definition. The word "adalimumab" is spelled the same whether the model said "add-al-EYE-mu-mab," "add-a-lee-MOO-mab," or something the caller had to ask you to repeat.

The tell in most eval suites: the test call passes, the ASR round-trip returns the expected string, and the customer still emails to say "why does your bot pronounce our product like that?" You are testing the LLM. You are not testing what the caller hears.

An honest pronunciation test has to score the acoustic layer, on a real audio path, on the exact model and voice combination that ships to production, and it has to keep doing so every time any of those pieces changes.

A working pronunciation-test setup

Concrete practices that work. None of them are exotic, but almost no team runs the full loop.

1. Build a pronunciation golden set

Extract every high-stakes term the agent will ever say, split by source:

• Static: your company name, product names, plan names, physical locations you own. These are constants.
• Dynamic from prompts: brand names, drug names, address components, disclaimer phrases the LLM will produce verbatim.
• Dynamic from tools: strings that come back from CRMs, EHRs, payment providers, order systems. Order numbers, provider names, plan IDs, medication names.

Each entry gets an expected pronunciation, tagged by language and voice profile. Treat this as source of truth checked into the repo, not a Notion page. When someone adds a new drug to your formulary or a new brand to your catalog, the pronunciation entry lands in the same PR.

2. Test the model x voice x dictionary matrix, not just one cell

This is the constraint most teams get wrong. Because rule support depends on both provider and model version, you need coverage across every combination you actually ship: elevenlabs/eleven_flash_v2_5, elevenlabs/eleven_v3, cartesia/sonic-3, cartesia/sonic-3.5, per language. When a rule format is not supported on a target, that is a test failure, not a config note.

The Famulor pronunciation guide frames it well: classify each problem word by whether it is said by the caller and misheard (STT) or said by the agent and mispronounced (TTS). Many terms, including your own company name, need both fixes at once. Simple cases only need an alias rule; proper nouns with no obvious standard pronunciation need a phoneme rule written in IPA. Your test matrix has to cover both sides for the same term.

3. Score the audio, not the text

Two workable approaches, either alone or combined:

• Snapshot regression: for each golden phrase on each voice, store a reference audio render, and diff against it after every model, voice, or dictionary change. The Future AGI writeup on ElevenLabs vs Cartesia recommends exactly this pattern, storing a reference audio render of each named phrase (brand names, product names, numbers, dates) and flagging drift after a provider change.
• Model-scored pronunciation: run each generated phrase through an audio-native scorer that outputs a pronunciation score against the expected phonetic form, and gate the deploy on a threshold.

Snapshot diffing catches drift. Model scoring catches wrongness. You want both.

4. Test in the LLM's own words, not fixtures

Half of pronunciation bugs never fire on unit-style fixtures because the LLM never actually says the phrase in the exact form you patched. It says "Doctor Smith at Saint Mary's" when your dictionary is keyed on "Dr. Smith at St. Mary's," and the alias never triggers. Vapi's docs are explicit that pronunciation dictionary searches are case-sensitive and rules apply in order, so the first matching rule wins and everything after it is dead code.

The fix is to run pronunciation tests as full conversational simulations, not synthetic TTS renders. Drive the agent with a persona that steers it into saying every high-stakes term at least once, capture the audio, and score that. Anything less and you are only testing the phrases the LLM would have said the same way regardless.

5. Regression-test on every trigger

At minimum, re-run the pronunciation suite on:

• Prompt changes that introduce new named entities.
• Model swaps (STT, LLM, or TTS).
• Voice or voice-version changes.
• Dictionary edits (both directions, in case a new rule breaks an old term).
• Any TTS provider version rev, including snapshot models like sonic-3-2026-01-12.

The pattern is identical to visual-regression testing in front-end work. Nobody trusts a UI diff to a human reviewer; nobody should trust a pronunciation diff to one either.

Where Roark fits

This is exactly the loop Roark is built to run. Roark's simulation testing dials your agent over real phone calls, so the audio path you evaluate is the one the caller actually hears, not a synthetic render into a WAV file. Personas can be scripted to push the conversation through your high-stakes phrases, and simulations run across 45 languages and accents so cross-lingual proper nouns get exercised on the same suite.

Roark's audio-native metrics score the sound of the call, not just the transcript. Pronunciation is one of the built-in dimensions, alongside pace, pauses, interruptions, emotion, and vocal stress. When a metric fails, Roark files it as an issue automatically, so a voice change that silently mispronounces "adalimumab" surfaces as a failed check on the specific turn, with the audio attached.

The regression loop closes with production call replay: the calls where a caller actually said "can you spell that?" get captured, and you can replay them against your updated agent to confirm the new dictionary or voice version actually fixes the problem, without waiting for another live caller to trip it. That is the difference between hoping the fix worked and knowing it did.

What to take away

Pronunciation failures are quiet. They do not show up in transcripts, they do not fail LLM evals, and they almost always ship on the back of a change that looked innocuous: a voice update, a dictionary edit, a model snapshot rev. The teams that hold pronunciation quality across model swaps are not the ones with the fanciest dictionaries. They are the ones with an acceptance suite that scores audio, runs on every change, and covers the full model, voice, and language matrix they actually deploy.

Text-only tests will keep telling you the agent is fine. Only audio tests will tell you what the caller hears.