All field notes

Voice AI Testing

·

Testing warm transfers in voice AI agents

The handoff from your voice agent to a human is where calls quietly go wrong. Here's how to test warm transfers before they ship, and score them in production.

James Zammit

James Zammit

Co-founder & CEO @ Roark

9 min read
Testing warm transfers in voice AI agents

Two voice AI product launches in the last two weeks share the same pitch. Convoso's no-code Voice AI Agent went generally available on November 2, built around engaging and qualifying leads before transferring sales-ready prospects to human agents. Three days later, XTIUM rolled out XTAI Voice with AI agents that answer every call, handle common service requests, and elevate complex interactions to skilled human agents. The marketing copy is almost interchangeable, and it points at the same bet: the AI runs the first 80% of the call, and a human takes the hard part.

Which means the handoff is the product. And the handoff, by every public account, is where voice agents quietly fall apart.

The handoff is a mini-call of its own

A warm transfer is not a single primitive. It is a sequence of distinct events that all have to go right in order: the agent recognises the trigger, says a plausible farewell to the caller, places a second outbound leg to the human, keeps the caller on hold without disconnecting them, delivers a private briefing the caller never hears, waits for the human to accept, bridges the two legs, and drops itself off the call. If any step misfires, the caller hears either dead silence, a half-told story repeated to a confused rep, or the dial tone.

Every major voice platform has a slightly different way of doing this, which makes it worse. On Vapi, assistant-based warm transfer "confirms that a person accepted before connecting the caller", and the platform exposes five traditional warm-transfer modes on Twilio calls plus an experimental assistant-based mode. On LiveKit, the warm transfer workflow spins up a private consultation room where the AI briefs the human before merging the caller in, orchestrated by a prebuilt WarmTransferTask. On Retell, the agentic warm transfer hands the transfer agent the original conversation and dynamic variables but, per NextLevel's comparison, nothing said after the transfer starts. Telnyx exposes a conferenced warm transfer pattern using Invite plus Skip Turn.

The implementations diverge on exactly the details that break in production: who holds the caller, what the human hears, whether the AI can come back if nobody answers, and what context the human receives. The QA plan has to be written against the platform you actually use, not a textbook flowchart.

Warm transfer steps that each have to go right
Warm transfer steps that each have to go right

The six failure modes you have to test for

Warm transfer failures tend to cluster into the same six patterns. Each one is testable, and each one needs its own scenario in a pre-launch suite.

1. The agent announces the transfer but never calls the tool. The model generates the farewell speech and then forgets to invoke the transfer. Vapi's documented pattern is that the prompt must require the acknowledgment and the tool invocation in the same response. If it does not, the caller hears "I'm connecting you now, one moment" followed by whatever the model decides to say next. Test: for every escalation trigger, assert the transfer tool call is actually emitted in the same turn as the farewell.

2. The handoff fires on the wrong intent. The caller says "I guess I could speak to someone" as a figure of speech, and the agent transfers them. Or they explicitly ask for a human and the agent tries one more round of self-help. Both fail your transfer-precision metric: of the calls the agent transferred, how many genuinely needed a human. Test both directions: scenarios where the caller insists, scenarios where they idly mention a human.

3. The caller is dropped during the proxy call. During the briefing, the agent is holding two concurrent legs, the caller and the human, plus the seconds it spends delivering the private briefing. The hold leg can die silently if the framework loses track of it, or if the hold music is never started. Test by instrumenting the caller side for dead air exceeding, say, eight seconds, and separately for an unintended disconnect.

4. The private briefing leaks. The AI thinks it is talking to the human. The caller hears everything. This happens when audio is re-routed incorrectly, or when the prompt does not clearly distinguish the two legs. Test by capturing both legs' audio and asserting the briefing text is present on the human leg and absent from the caller leg.

5. Nobody answers. The human line rings out, and the agent needs to come back. LiveKit's warm transfer workflow handles this explicitly: the WarmTransferTask exposes a ringing_timeout controlling how long to wait for the human to answer, and if the human hangs up the caller conversation resumes. Many homebrewed implementations just drop the caller. Test every no-answer and busy variant, and assert the agent either returns to the caller gracefully or follows a documented fallback (voicemail, callback, queue).

6. Context does not reach the human. The caller spent four minutes explaining their situation. The human picks up and gets "customer has an issue." Vapi's traditional modes say a message of up to 1,000 characters or a summary, and the content of that summary is on your team to design. Test by scoring the briefing against the actual facts of the simulated call: did it name the customer, state the intent, surface the one piece of information the human needs to open with?

Why standard QA keeps missing the handoff

The usual reason teams do not catch these in staging is mundane: the handoff requires two phone legs, which means two phone numbers, two testers, and a lot of coordination. So nobody tests it at the volume the live product gets. The transfer tool gets a unit test, the private briefing gets eyeballed once, and the compound behavior, which is the actual product, ships without a regression net.

The deeper reason is that the handoff happens at the exact boundary where most text-based evaluation gives up. A transcript eval cannot tell you whether the caller heard the private briefing. It cannot tell you whether there were six seconds of dead air between the farewell and the hold music. It cannot distinguish a clean SIP REFER from a conferenced bridge that forgot to drop the AI. The failure signatures live in the audio, not the tokens.

This is where Roark's simulation-first approach lands. Roark dials your agent over real telephony, not a text loopback, and its personas are built from distinct voices, accents, pace, emotional register, and background-noise environments. A "frustrated customer who asks for a manager after two denials" is a persona, not a prompt string. You run a batch of them against the warm-transfer path and score every attempt on audio-native metrics: did the caller hear dead air, did the agent interrupt the human's greeting, did the handoff land inside your acceptable duration window.

An illustrative transfer suite run before shipping a new handoff rule
An illustrative transfer suite run before shipping a new handoff rule

A transfer suite you can run before every release

A useful pre-launch transfer suite covers the full product surface, not just the happy path. Here is the shape of one that holds up for most B2B agents.

  • Explicit asks. "Can I speak to a human," in five tones (neutral, frustrated, polite, confused, angry). Assert the transfer tool fires, the farewell is appropriate to tone, and the briefing names the issue.
  • Implicit asks. "I need to talk to someone about this," "this is not what I called for." Assert the transfer tool fires without the agent trying to re-solve first.
  • False positives. "I was going to call my husband about this," "I spoke to someone there last week." Assert the agent does not transfer.
  • Edge triggers. Caller mentions a restricted topic (legal threat, self-harm, a regulated domain). Assert the agent follows your escalation policy, not the default.
  • Human unavailable. Scenarios where the destination line rings out, is busy, or refuses the call. Assert the agent returns to the caller with a working fallback.
  • Mid-transfer interruption. The caller barges in during the farewell or during the hold music. Assert the agent handles the interruption without dropping either leg.
  • Context fidelity. For every successful handoff, score the private briefing against the facts of the call. The caller's name, the account identifier, the question, and the one thing the human needs to say first.

Every scenario produces a pass/fail and a score on the metrics you care about. Every failure becomes a replayable regression test for the next release. That last part matters more than it sounds: the fastest way to regress a handoff is to tune the trigger prompt for one failure mode and accidentally break two others. If you do not have the earlier failures replaying against every build, you will ship the regression and find out from a customer.

What to score on live calls

Pre-launch tests cover known scenarios. Production will find the ones you did not think of. The metrics worth filing issues against for every real transfer:

  • Transfer precision. Of the calls where the agent transferred, how many needed a human. Measured by reviewing the preceding transcript against your definition of "needed."
  • Transfer recall. Of the calls where the caller was pushed back to voicemail or hung up frustrated, how many should have been transferred. The harder number to measure, and the more useful one.
  • Pre-handoff latency. Time between the escalation trigger and the start of the farewell. A caller who waits three seconds for "one moment" hears a broken agent.
  • Hold-side dead air. Seconds of silence on the caller leg during the briefing. Should be covered by hold music, tone, or an occasional check-in.
  • Briefing coverage. Does the private message to the human contain the caller name, account, intent, and one key fact. Scored per call, aggregated per cohort.
  • Return-to-caller rate. When the human does not answer, does the agent come back cleanly, or does the caller get dropped.
  • Disclosure retention. If you were required to disclose the agent was AI, does the disclosure travel with the transfer, or does the human inherit a call where nobody told the caller they were about to speak to an AI's handoff queue.
A scored handoff where the briefing landed but the hold-side had five seconds of dead air
A scored handoff where the briefing landed but the hold-side had five seconds of dead air

Roark scores every live call against a suite like this, files an issue automatically when a metric or check breaks, and lets you capture that real failure as a replayable scenario against the next build. Transcripts and OpenTelemetry traces sit under each issue, so when the transfer breaks at the SIP layer the span is already there. One-click integrations for Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs mean the call capture is the same regardless of which transfer primitive you use.

The handoff is where callers decide whether the agent was any good

Every launch pitch for a voice AI product promises the agent will handle the easy cases and send the hard ones to a human. Convoso's and XTIUM's launches in the last two weeks are just the most recent examples. That promise is only as good as the moment where it is kept. If the handoff is noisy, the caller remembers the whole call as noisy. If the briefing is empty, the human restarts the conversation and the caller tells their boss the AI wasted their time.

Test it like it is a product. Because for the caller, it is the only part they will tell someone else about.

James Zammit

Written by

James Zammit · Co-founder & CEO @ Roark

Building Roark — the quality platform that simulates, monitors, and auto-improves voice and chat agents.

Bring a recording.
We’ll score it live.

See your own agent measured on the audio it actually produced, in the demo, in real time. Stop guessing whether your voice AI works.

Or get $50 in credit with a work email · read the docs · support@roark.ai