Barge-in and endpointing get all the attention in turn-taking. Backchannels get none, and they are the failure mode your callers actually notice. A caller says "uh-huh" while your agent is halfway through explaining a policy, the agent aborts its own TTS, and the next thing the caller hears is either dead air or the agent restarting the sentence they already interrupted themselves on. The transcript looks fine. The call is broken.
This post is a test plan for backchannel handling in voice AI agents: what backchannels are, why every major stack now treats them as a distinct signal, and the specific scenarios you should be running against every candidate build. If you build on Vapi, LiveKit, Pipecat, Retell, or ElevenLabs, you already own a turn-detection pipeline. Whether it handles "mhm" correctly is a testable question, and the tests are cheap to build.
Backchannels are not interruptions
A backchannel is a continuer. In the conversation-analysis literature that goes back to Schegloff, tokens like "uh-huh," "mhm," "yeah," and "right" are the listener's way of telling the speaker to keep going. They are not bids for the floor.
Voice agents built on pure VAD cannot make that distinction. As one 2026 implementation guide puts it, pure VAD treats backchanneling as either silence or a full barge-in attempt, and neither is right. The production stack has been migrating toward dedicated turn-taking models that classify backchannel vs. barge-in vs. continued silence as a learned signal instead of an energy threshold.
Deepgram put numbers on how bad the default is. In a scored evaluation against 30 hours of hand-annotated conversation, an energy-threshold detector fired on 45% of the backchannels and non-speech noise it heard, climbing to 52% in casual talk where backchannels are densest. None of those triggers was a bid to speak, and every one would stop a voice agent mid-sentence.
There are two failure modes to test for, and they are opposites.
- The false interrupt. The caller mhm's while listening. The agent stops speaking, restarts, or worse, treats the mhm as an answer to the last question it asked ("did you want to add rental to that quote?" then "mhm" then agent books rental).
- The missed floor bid. The caller says "uh-huh, but actually the address is wrong." The token starts as a continuer and turns into a real turn. If the agent's policy is "backchannels never interrupt," it happily keeps talking over the correction.
Both are testable. Neither shows up in aggregate latency dashboards.
What the platforms are shipping
The pattern across every serious stack is the same: split the decision. VAD detects sound. A separate model or heuristic decides whether the sound was a turn.
LiveKit shipped Turn Detector v1 in June. The models listen to the user's speech directly instead of waiting on a transcript, fusing semantic and acoustic understanding into a single end-of-turn prediction, and combine semantic understanding with acoustic cues like intonation, pitch, and rhythm to reach state-of-the-art end-of-turn accuracy without relying on a transcript.
Pipecat has been iterating on Smart Turn v2 on the audio-first side. NVIDIA's NeMo Labs Voice Agent went straight at backchannels with a shipped phrase list of 78 tokens including "uh huh," "mhmm," "okay," "right," "sure," and "i see." When one of those phrases is matched while the bot is speaking, the service pushes it upstream as a transcription frame wrapped in parentheses ("(uh huh)") that reaches the client transcript and audio logger without entering the LLM context or triggering an interruption. That is a real, working policy: recognize the token, log it, keep speaking, and do not let it poison the LLM context.
Deepgram's take leans on the transcript: the specific tokens ("uh-huh," "mhmm") land in the transcript once filler words are on, and handling them becomes a lookup. The negative forms ("uh-uh," "nuh-uh," "mm-mm") are the exception, and they matter, because "uh-uh" (no) is a real answer that a naive backchannel filter will drop on the floor.
The point is not that any of these approaches is complete. It is that every stack now exposes at least three knobs that interact: the VAD threshold, the turn model or classifier, and whatever platform-level timer sits on top. As AssemblyAI's own guide puts it, the common failure is a platform-level timer sitting on top of a model that's already made a better decision. Every configuration change to those knobs demands a regression run.
The test surface
A backchannel test suite is small. Maybe fifteen scenarios per language you support, run in real audio, with the agent talking a substantive turn so the backchannel has something to interrupt. The categories that matter:
Standalone affirmative continuers. The caller says "mhm" once, mid-agent-turn, and stays silent. Expected behavior: the agent keeps speaking. Do not commit the turn. Do not append "mhm" to the LLM context as a user message. Score: pass if TTS continues uninterrupted and no LLM turn fires.
Clusters. Three or four "uh-huh"s across a thirty-second agent explanation. Same expected behavior, but this is where naive filters fail: they suppress the first two and then treat the third as barge-in when a debounce window resets.
Negative tokens. The caller says "uh-uh" or "mm-mm" in response to a yes/no question. This is not a backchannel. It is the answer. Any agent that suppresses "uh-uh" as a continuer will silently misroute the call.
Continuer that becomes a floor bid. "Uh-huh, but actually the address is wrong." Deepgram's rule is to wait briefly after the token to see whether the caller adds more words; word end-timestamps plus a short wait window give you this distinction, the practical substitute for the pitch contour you can't retrieve. Test that the agent yields within a bounded window once the follow-on words arrive.
Ambient noise that looks like a continuer. A door closing, a background "yeah" from a TV, a cough. Energy VAD will trigger. A well-tuned pipeline will not. Score the false-interrupt rate.
Real barge-in, not a backchannel. Regression control. "Wait, stop, that's the wrong policy." The agent must yield. If backchannel suppression is too aggressive, real interrupts get eaten too, and you will not notice until a call ends badly.
Accent and cadence variation. Backchannel tokens are language-specific and phonetically variable. A US "uh-huh" is not a UK "mm" is not a South-Indian "haan." Any regression suite that only tests one variant is a regression suite that will pass in staging and fail in production.
What to measure
Four numbers keep the pipeline honest.
- False-barge-in rate on backchannels. Percentage of scored backchannel events where the agent aborted TTS. Target: near zero on affirmative continuers, measured separately from ambient-noise false barge-in.
- Missed floor-bid rate. Percentage of continuer-turned-floor-bid scenarios where the agent kept talking past the follow-on words. Measured in the same suite as the above; the two failure modes trade off against each other.
- Context-poisoning rate. Percentage of backchannel events that leaked into the LLM's user-turn context. NeMo's model of writing them into the audit transcript but not the LLM context is the correct policy; any agent that lets "mhm" become a user turn will eventually take it as an answer.
- Real-barge-in preservation. The regression control. If this drops when the backchannel filter tightens, you are trading one failure mode for another.
One writeup on barge-in metrics notes that barge-in behaviour is difficult to evaluate from call logs alone, and that a false-barge-in rate above 8% means your VAD threshold is too sensitive. Backchannel handling is the same problem with a sharper edge, because the "meaningful caller speech" test alone will not catch the continuer that never became a floor bid.
Building the runs
There are only two ways to test this seriously.
The bad way: ship it, listen to production for a week, ticket the ones that went wrong, argue with the platform vendor about whose knob is at fault, tune, repeat. This is how most teams do it.
The tolerable way: build a small persona set that produces backchannels on cue during a real phone call. The scenarios above become fifteen or so run configurations. Each persona has a script and a cadence: "listen through the agent's explanation, mhm at 8s, mhm at 14s, stay silent," or "listen to the price recap, then say 'uh-huh, but that's the wrong billing address.'" Every candidate build gets dialed by every persona, over real telephony, with the resulting audio scored.
The reason it has to be real telephony is codec artifacts. VAD and backchannel classifiers behave differently on 8 kHz G.711 than on a clean loopback, and the whole point of the suite is to catch failures that only happen on real calls.
Where Roark fits
This is exactly what Roark is built for. Roark's simulation testing dials your agent over real PSTN and WebRTC, so backchannel scenarios run against the same codec path production uses. Personas define the caller: voice, accent, pace, and emotional register, plus background environment. That is the axis backchannels live on, and it is the axis a text-loopback test bench can never cover. Roark simulates and scores in 45 languages and accents, so the "US uh-huh vs UK mm vs Indian haan" problem becomes a persona set, not a manual QA task.
Every simulated call is scored on Roark's audio-native metrics. The models score the sound of the call, not just the transcript: interruptions, pace and pauses, pronunciation, emotion. A false interrupt on a backchannel is scored as an interruption event you can filter and count; a missed floor bid shows up as caller speech the agent talked over. What breaks is filed as an issue automatically.
Two more pieces close the loop. First, production call replay: when a real call goes sideways because the agent aborted on an "mhm," capture that call and replay it against your next build. Real failures become regression tests you never write by hand. Second, the same suite you built for pre-launch runs continuously against production traffic, so a model swap or a VAD threshold tweak that regresses backchannel handling files an issue on the first call it breaks, not the fiftieth.
Roark integrates one-click with Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs, so wiring the suite against whatever stack you already run is not the reason this project stalls. The docs walk the setup for each.
The failure you never see
The reason backchannels stay untested is that they do not look like failures in the log. The agent responded. The transcript is clean. The call ended. But the caller heard themselves get talked over three times, or heard the agent restart the same sentence, or hung up because they said "uh-uh" and the agent booked the appointment anyway.
Turn detection has come a long way in the last year. The models are good. What is missing is the discipline to test them on the specific inputs they were built to handle, on real audio, before the caller does. Build the fifteen scenarios. Run them on every candidate build. The suite pays for itself the first time a model swap silently regresses "mhm" handling and the issue files itself instead of the caller filing a complaint.
