Your voice agent works in dev. A tester calls the number from their laptop softphone, the agent asks for a four-digit PIN, they press the keys, the agent reads them back, the ticket closes. A week later a real customer calls from a real mobile carrier, presses the same four keys, and nothing happens. No error in the logs. The call just sits there until the agent times out and apologises.
This is the DTMF failure mode that never appears in staging, and it is showing up more often now that voice agents are moving into flows that actually need keypad input. Assort Health just raised a $120M Series C to expand voice AI across the patient journey, and voice agents on OpenAI's Realtime stack are closing sales end to end on retail sites. Patient intake, account authentication, and payments all rely on the caller pressing digits reliably. When that step is silently broken, the rest of the agent does not matter.
This piece is about testing inbound DTMF: the tones your caller sends to your agent, not the ones your agent sends to a downstream IVR. It is a different problem from outbound phone-tree navigation, and it fails in different places.
Three modes, one call
DTMF looks like a solved problem until you inspect the wire. Voice AI platforms have to accept keypad input across three encodings, and only one of them is fully robust in modern deployments.
In-band. The actual audio tones live inside the audio stream. This is what a physical telephone does. It works fine on PSTN with G.711 but degrades badly when the audio is compressed or routed through WebRTC. Twilio's <Gather> verb, for example, does not detect in-band DTMF at all: any in-band tones are forwarded unmodified to other parties and never surfaced as digit events.
RFC 2833 (also RFC 4733). DTMF events are sent as named telephone-event RTP packets alongside the audio, independent of the codec. This is what modern SIP carriers default to, and it is what Twilio's SIP Interface supports for send and receive. On the Vapi side, the built-in DTMF tool uses out-of-band RFC 2833 because in-band suffers from codec-quality issues.
SIP INFO. DTMF sent as SIP signaling messages, out of the media path entirely. Historically used by older Cisco and Avaya PBX systems. Twilio does not currently support SIP INFO DTMF, and neither does its <Gather> verb.
WebRTC. Its own failure mode. Ultravox's docs are blunt about it: due to the audio codec used in WebRTC connections, DTMF tones are inaudible when using WebRTC. A browser-based test call cannot exercise the same code path a phone call does.
The trap is that all three modes negotiate cleanly in SDP. The call connects. The audio flows. Keypad presses just produce no response, with no error in the logs. One engineering post traces this exact pattern to a 34% abandonment rate on the PIN entry step of a financial services deployment.
What actually breaks
Assuming the transport mode is right, DTMF still has plenty of ways to fail on inbound. Test suites need to cover each one deliberately.
Mode mismatch. Your platform is configured for RFC 2833; the carrier sends in-band. Or your imported SIP trunk sends SIP INFO to a platform that only listens for RFC 2833. Fix: match the mode on both sides, and prefer PCMU over compressed codecs for DTMF-critical routes.
Timing and terminators. The keypad input plan defines how many digits to collect, how long to wait between digits, and which character terminates the sequence. Vapi's keypadInputPlan, for example, exposes timeoutSeconds and delimiters and adds the collected digits to the conversation as a user message. Get the timeout wrong and callers finish keying in an account number just after the agent has already given up and moved on.
Multi-digit collection. A four-digit PIN, an eight-digit account number, an SSN. If the platform emits one event per key press, your prompt has to buffer them; if it emits one event per completed sequence, the delimiter matters. Microsoft's Copilot Studio distinguishes single-digit dial-pad input from a batched /DTMFkey 123 command in its test panel, and warns that the chat test does not validate DTMF length, timers, term key, or caching.
No-input reprompt. The caller hesitated, or the carrier ate the first tone, or the codec compressed it away. Copilot Studio's DTMF flow explicitly repeats the question if no key matches, then falls through to an "unknown dial pad press" topic. If your agent skips the reprompt and treats "no input" as "invalid," you will hang up on hesitant callers.
Mixed modality. A caller who spoke their name, keyed their PIN, and now wants to speak their question. Or a caller who was told to press 1 for English, said "one" out loud, and did not press anything. Ozonetel and others argue for DTMF entry as the default for PINs, OTPs, and payment authorisations precisely because ASR ambiguity fails on high-accuracy fields, but the handoff between voice and keypad has to be tested end to end.
Speech masquerading as keypad. The agent's DTMF tool is invoked, digits get logged as {"digits": 1}, everything looks fine, and the downstream system reacts as if nothing happened. Vapi's community threads are full of variants of this pattern, usually resolved by checking the telephony provider's DTMF-relay setting or the SIP trunk configuration.
None of these fail in a text-based test harness. Most do not fail on a laptop softphone. They fail on real phones over real carriers.
The tests your CI is missing
If your DTMF coverage is a QA engineer with a mobile phone and a checklist, it will not survive a launch. The tests that catch this class of bug have to run over real telephony, on the specific carrier and codec path production uses, on every change to the agent or platform config.
A useful DTMF suite for an inbound agent covers, at minimum:
- Happy path per digit length. Four-digit PIN, ten-digit phone number, sixteen-digit card number. One test each, with the correct digits keyed in at natural spacing.
- Timeout on partial input. Caller enters two of four digits and stops. The agent should reprompt, not fail out.
- Terminator behaviour. For flows that use
#or*as a delimiter, an input with and without the terminator. Half of the real-world failures are callers who never press the terminator. - Inter-digit spacing. Fast keying (bank card entry) and slow keying (someone reading digits off a card while driving). Some platforms accept inputs more reliably at wider spacing.
- Wrong digit, then correct. Caller keys one wrong digit and self-corrects. Does the agent accept the correction, or is the buffer poisoned?
- Speech during keypad prompt. Caller says "seven eight two nine" instead of pressing. Does ASR pick it up? Does the DTMF timeout still fire when it should not?
- No input at all. Silence. The reprompt path.
- Mode-mismatch canary. A test call from a route that intentionally sends in-band DTMF, to prove the RFC 2833 path fails loudly instead of silently. If this test passes when it should not, something in your stack is falling back to in-band decode.
The reason these tests are hard to run manually is not that any single one is complex. It is that DTMF failures are transport-dependent and combinatorial. Change a codec preference, swap a SIP trunk, upgrade a platform SDK, and the failure surface shifts under you. Sample-based QA cannot cover that.
Mixed voice and keypad, the real failure surface
Every DTMF-heavy flow eventually becomes a mixed-modality flow. The caller speaks a preferred language, presses digits for their PIN, speaks their question, then presses to confirm. Testing each modality in isolation catches maybe half the real failures. The rest live at the boundaries.
Two patterns to test explicitly:
Voice-then-keypad handoff. The agent finishes a spoken exchange, prompts for keypad input, and waits. What is the endpointing behaviour during the DTMF window? If the agent still treats the mic as hot and interprets breath noise as a barge-in, you will get spurious ASR events on top of the keypad stream. If it goes fully silent, callers do not know it is listening. The design pattern most platforms recommend is a slower response cadence for machine-like exchanges, then faster for human conversation.
Keypad-then-voice recovery. Digits captured, downstream lookup fires, result comes back, agent resumes conversation. The transition needs to preserve context. In banking flows, step-up authentication mid-conversation is a common pattern: the caller starts with a general question, escalates to a payment intent, and the agent moves from a low-trust to a high-trust state without restarting the call. That transition is a testable regression, and it is almost never covered in a single-modality test suite.
Turning production failures into regression tests
The DTMF bugs that make it to production are the ones nobody could reproduce in staging. That is what makes them worth capturing rather than just closing.
This is where Roark fits in. Roark's simulations dial your agent over real PSTN and WebRTC, which means the same transport where DTMF-mode mismatches actually fail is the transport the tests run on. You can define personas that represent the callers who hit these flows, in the accents and background environments they call from, and schedule the suite to run on every agent update. Roark has one-click integrations for Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs, so wiring the suite to CI does not require a bespoke harness.
For production, every live call is scored against the metric suite, and what breaks is filed as an issue. A caller who abandons on the PIN prompt is not just a call-quality data point; it is a failed check that surfaces automatically. And Roark's production call replay lets you take that real failed call and replay it against updated agent logic, turning a one-off customer complaint into a repeatable regression test for the DTMF path.
For regulated flows, the reporting side matters too. Roark is SOC 2 Type II certified and offers a HIPAA BAA, which is table stakes for healthcare voice agents doing intake by keypad. OTEL traces are exposed on every call, so DTMF timing lives in the same tracing surface as the rest of the agent.
The takeaway
Inbound DTMF is one of the most overlooked configuration surfaces in voice AI. It is easy to get right in a demo, easy to get wrong in production, and almost impossible to test with a laptop softphone. The teams that ship regulated voice agents (healthcare, banking, insurance, anything that touches payments) will run into every failure mode above, usually one at a time, usually in production.
Build the DTMF test suite before launch. Run it over real telephony. Make it part of the same regression cadence you run against your prompt changes, model swaps, and provider updates. The alternative is finding out at 3am, from a customer, that your PIN prompt has been silently dropping tones on one carrier for a week.
