Testing alphanumeric capture and readback in voice AI agents

When OpenAI shipped gpt-realtime, one of the capabilities they called out explicitly was that the model was better at "repeating back alphanumerics" on support calls. A year on, the follow-up gpt-realtime-2 landed with "configurable reasoning for speech-to-speech agents" and vendor benchmarks showing double-digit gains on audio instruction following. That is real progress. It is also a warning label.

The reason vendors keep advertising alphanumeric readback is that alphanumeric capture is where voice agents visibly fall apart in production. It is not an ASR problem you fix by upgrading a model. It is a class of failure that requires its own regression suite, its own metrics, and its own personas. If your agent takes an order number, a policy ID, a member ID, a confirmation code, or an email address over the phone, you need to test that path the way you test payments, not the way you test small talk.

Why alphanumeric strings break where prose doesn't

Normal conversation is easy for a modern speech pipeline because language models carry an enormous prior over what words follow which. Alphanumerics do not have that. As AssemblyAI put it in a recent post on voice-agent accuracy, alphanumeric strings are unusually brittle because acoustic similarity between letters (B/V/P, D/T, M/N) and between digits (5/9, 4/14) creates confusion, and there is no semantic context for the model to fall back on. The model gets it right or it doesn't.

Speechmatics made the same point more concretely: SKUs, model numbers and booking references are arbitrary strings, not natural language, and letter and digit names are acoustically confusable at telephone bandwidth. "XL" and "XS" sound alike over a phone. "B15D" and "P15D" sound alike over a phone. This is not a bug that gets fixed by a bigger LLM sitting downstream, because the pipeline has already committed to the wrong character by the time the LLM sees the transcript.

Deepgram's write-up on conversation context makes the same observation from the ASR side: errors cluster in letter-by-letter spellings and short replies, including one-word confirmations or single digits. One misheard character in a nine-character code cascades. The agent reads back the wrong number, the caller corrects it, and two more turns go to a repair loop that should never have started.

The failure taxonomy nobody trains against

If you look at the EVA-Bench audit of production voice agents, the alphanumeric failure modes are surprisingly uniform across vendors. The paper documents entity mispronunciation as the dominant failure class across every model tested: substitutions, omissions, spurious insertions, and phonetic confusions between similar-sounding characters. GPT-Realtime-1.5 swaps C and P inside facility codes. Aura-2 substitutes L for I in a medical license number. Ultravox-Realtime misreads a confirmation code by turning an "8" into an "I". Voxtral-4B-TTS silently appends a trailing digit to a request identifier. Kokoro mispronounces "SK130" as "S-Cone 130".

These are not one-off hallucinations. They are the same acoustic-similarity failures the NATO phonetic alphabet was designed to solve seventy years ago, appearing in an LLM-driven pipeline. The ICAO code words were selected specifically because higher noise levels do not create new confusions, they intensify the confusions already inherent between letters. Modern realtime models have not repealed that finding. They inherited it.

The taxonomy your regression suite needs to cover:

• Substitution. One character replaced by an acoustically or visually similar one. The classic B/D/P/V, M/N, S/F, and 5/9, 4/14 pairs, plus vendor-specific patterns (C→P, Z→V, L→I, 8→I).
• Omission. A character silently dropped, most often the first or last in a run.
• Insertion. An extra digit or letter appended, often mid-transition between number and letter.
• Segmentation. "SK130" heard as "S-Cone 130", or "one-eight-hundred" transcribed as "1 800" then rendered as "eighteen hundred" on readback.
• Phonetic-alphabet failure. The caller says "N as in November" and the agent captures both letters, or only "November", or nothing.

Every one of those is a distinct assertion in a good test suite. "The identifier matches" is not enough; you want to know which failure mode you regressed into so you can decide whether to bias the ASR, change the prompt, or fall back to DTMF.

Word error rate is the wrong metric

The bluntest way to get this wrong is to score alphanumeric capture with WER. Speechmatics' benchmark is a good illustration: 96.6% word-level accuracy can still mean only 77% identifier exact match. If a caller's confirmation code is fifteen characters and any single one is off, the whole call fails, even though the transcript is 93% right. WER averages you into the ditch.

The metrics you want, side by side:

Readback fidelity is the sneaky one. Plenty of agents capture the identifier correctly, then their TTS mangles the readback ("A" pronounced as "uh", digits run together, "zero" swapped for "oh") and the caller confirms something that isn't actually what got stored. Or the opposite: the agent captured "P15D" and read back "B15D", the caller heard what they meant to say and confirmed. Now you have a silent, confidently-wrong record. Text-based evals miss both patterns entirely, because the transcript looks fine and the caller said "yes".

What a real alphanumeric test suite covers

Assume your production traffic contains at least a few of: account numbers, member IDs, policy numbers, order confirmations, tracking numbers, appointment codes, email addresses, ZIP or postal codes, or license plate numbers. Every one of those wants its own scenario matrix.

For each identifier type, vary:

1. Length. 4-character short codes, 8 to 12-character normal IDs, 16 to 20-character long codes. Failure rates are not linear in length; there is usually a cliff.
2. Composition. Digits only, letters only, mixed, with hyphens or dashes, with prefixes ("REQ-", "P-"). Mixed strings are where the segmentation failures live.
3. Speaking style. Digit-by-digit ("four, five, eight, two"), grouped ("forty-five, eighty-two"), spelled with a phonetic alphabet ("Papa four five eight two Quebec"), and one-shot fast ("P4582Q").
4. Caller pace and register. Normal, fast, slow with pauses, hesitant with self-corrections ("four, five, uh, no, four, six...").
5. Confusable pairs planted. Codes that force B vs D, 5 vs 9, F vs S, M vs N. If your suite never contains a "B4D9M" it will never regress on it.
6. Accent and language. Speechmatics notes that ASR systems generalize poorly on accented speech, with accuracy typically dropping around 15% on strong regional accents. Include the accents in your actual call logs, not just the ones your STT vendor benchmarks against.
7. Channel. Real telephony audio, not clean studio recordings. The narrowband codec is where "XL" becomes "XS".
8. Noise. Kitchen, car, office. Not to make it impossible, but to make sure the intensified-confusion effect the NATO researchers documented shows up in your dashboard before it shows up in a complaint.

Then, for each scenario, the assertion is a small set: captured value equals the ground truth; readback text equals the captured value; readback audio, transcribed back, equals the readback text; if a correction was injected, the final captured value reflects it. That is what "test alphanumeric capture" actually looks like when you write it out.

The recovery flow is the test that matters most

Every alphanumeric conversation has two paths: happy (captured right on the first try, readback confirmed, done) and unhappy (something got mis-captured, the caller corrects, the agent re-captures, re-reads back, and eventually gets there or gives up). The unhappy path is where you win or lose real revenue.

Things worth explicitly testing on the correction flow:

• The agent asks for a specific character position, not the whole string, when the caller says "the fourth one is wrong".
• The agent switches to a phonetic-alphabet mode when it gets two wrong captures in a row, not on the fifth attempt.
• The agent knows when to offer DTMF fallback, and knows how to prompt for it in a way the caller understands.
• The agent does not silently re-capture the same wrong character because its own biased ASR keeps producing the same mistake.
• The agent does not enter an infinite readback loop when the caller has said "yes" but with a rising, uncertain intonation.

The last two are audio-native failures. A transcript-based eval will report success on both. A human listening to the call will not.

Testing all of this with Roark

Here is the part where the honest answer is: you can build this yourself, and you should not. Every voice-AI team we work with started by writing a small internal harness for alphanumeric capture, and every one of them stalled at the same place: they could generate the strings, they could dial the agent, but they could not score the audio side of the readback loop without building an audio-native evaluator, and they could not scale personas without provisioning a dozen TTS voices and a real telephony path.

That is exactly the shape of the problem Roark solves. Simulation testing dials your agent over real phone calls, PSTN and WebRTC, using personas that define caller voice, accent, pace, and background noise across 45 languages and accents. You author scenarios per identifier type, define happy and unhappy correction flows, and Roark scores every run against 64+ built-in metrics plus any custom ones you write, on the audio itself, not just the transcript. Runs are recurring and can be triggered over HTTP, so an alphanumeric suite gates the same CI that gates a prompt change or a model swap.

The other half is what happens after launch. Every live call is scored against the same metrics, and when a real caller's confirmation code gets misheard, the failure is filed as an issue with the audio and the trace attached. That call, the real one, is then replayable as a regression test against your updated agent. In practice, this means your alphanumeric suite grows organically: the failures your callers hit in production become the tests that guard the next release, so you never regress into the same substitution twice.

Wire-up is one-click for Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs, or ingest via the Node or Python SDK if you run a custom stack. If you are shipping into a regulated vertical, Roark is SOC 2 Type II certified and a HIPAA BAA is available, so the same suite covers your healthcare or insurance member-ID scenarios.

A short launch checklist

Before your next prompt change or model swap touches production:

1. You have at least one scenario per identifier type your agent captures, and each varies length, composition, speaking style, accent, and channel.
2. You are scoring sequence exact match and readback fidelity, not WER.
3. Your suite includes deliberately confusable characters (B/D, 5/9, M/N, F/S) in the ground truth.
4. Your correction flow is tested as its own scenario, including "wrong character at position N" and "the caller says yes but sounds unsure".
5. Every real production failure on an identifier capture is turned into a replayable regression test, so you never ship the same bug twice.
6. The suite runs on every deploy, not once at launch.

The gpt-realtime family will keep getting better at reading back alphanumerics. So will its competitors. That is not the same as your agent getting better, and it is definitely not the same as your agent not regressing when you swap the underlying model. Test the readback loop like the money path it is.