On September 15, Google shipped Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking, with asynchronous function calling as a headline capability: the model executes tools and API calls in the background while continuing the conversation, so it can acknowledge a request and keep chatting while tasks finish. OpenAI's gpt-realtime introduced the same pattern last year, so long-running function calls no longer disrupt the flow of a session, and async tool calling in the Responses API is now supported by GPT-6 Astra and later.
This is the new default architecture. It's also a whole new class of ways to fail on a phone call, and most existing voice-agent test suites don't touch them.
What "async" actually means on a voice call
Three different vendor features get called "asynchronous" and they answer different questions. It's worth being precise, because your test scenarios differ for each.
The first, and the one this post is really about, is async function calling: the model dispatches a tool call, the platform keeps the audio stream open, and the model keeps generating speech while the tool runs. When the tool returns, the platform stitches the result back into the conversation. OpenAI's docs describe attaching a task_handle to each async tool so the model can later refer to it by name, and Google's Live API defaults to non-blocking execution so the models call tools in the background while the conversation continues.
The second is background response generation (OpenAI's Background mode): the whole response is generated asynchronously so a client can poll response objects to check status over time. This is a batch pattern; it's not what a live phone agent does.
The third is placeholder responses for pending calls. On synchronous models that can't keep talking, the GA Realtime API added tuned placeholders so if you ask the model for the results of a function call, it'll say something like "I'm still waiting on that". That's a UX bandage on a blocking call, not real concurrency.
For a phone agent in 2026, "async tool calls" almost always means the first one. The model is speaking while your tool is running. That is the source of the new failure modes.
The new failure modes
The old failure mode for a slow tool call was dead air. That one is largely solved now, and if you're still shipping filler phrases as a workaround, you're testing the wrong thing. The interesting failures with async tools are more subtle, and they show up as a plausible-sounding call that quietly went wrong.
Premature commitment. The agent narrates a value it doesn't have yet. It says "I believe it's $29" while lookup_price is still in flight, and when the tool returns $34 the correction lands two turns later. On a transcript this reads as a small stumble; on a recorded call it sounds like the agent hallucinated a price. As one developer commenting on Google's release put it, "what should the user hear if the tool times out, returns partial data, or completes after the conversation has moved on?". That is the design question, and it's a test target.
Audio-text divergence. Native speech-to-speech models produce text and audio at the same time, and they can say different things: the text might read "3:00 PM" while the audio says "3:30 PM". Async tool returns exacerbate this. When a result lands mid-utterance, the model may write one value into the transcript and speak another. A JSON log check will never see it.
Late arrival, wrong topic. The user asked about SKU 8842, then the agent moved them to a policy question. Ninety seconds later lookup_price returns. Does the agent volunteer the price into the middle of the policy answer? Wait for a natural break? Drop it? The right behavior depends on your product; the wrong behavior is undefined behavior.
Race with new caller input. The user gave one date, the tool started running against that date, then the user corrected themselves. When the result comes back, it's stale. The agent has to notice and re-issue.
Timeout ambiguity. Anthropic documents a four-minute pending-call timeout; neither OpenAI nor Google states a limit for an outstanding client call. Your application owns that policy. What the caller hears when your policy fires is your problem to test.
Barge-in over a pending call. The caller interrupts to ask "any update on that lookup?" while the task is still running. The agent needs to answer honestly ("still checking") without cancelling the task, without inventing a result, and without dropping into a scripted filler that pretends the tool doesn't exist.
Silent completion. The tool returns successfully, the agent processes the result, but never verbalizes it. You'll only see this in the audio. The state store thinks the job is done. The caller is still waiting.
None of these are hypothetical. OpenAI's own evaluation guide for GPT-Live acknowledges the same shape of problem: the model can listen, speak, receive corrections, and delegate work while audio continues, so evaluating only transcript or final responses misses important failure modes, and a natural-sounding confirmation does not prove the correct backend actions were completed.
What to actually test
A useful test suite for async tool calls is not "does the tool get called." It's a matrix of caller behavior crossed with tool timing. Here is a minimum viable set of scenarios worth codifying and running on every deploy.
- acknowledge_only_before_result — Caller asks for a value. The agent acknowledges but must not quote any number, date, or identifier until the tool returns. Scored on the audio, not the transcript.
- no_premature_commitment — Similar, but the caller pressures ("just ballpark it"). The agent should hold the line: no invented value, no hedged value that could be mistaken for the answer.
- late_arrival_topic_shifted — Force a topic change after dispatch. When the result lands, does the agent re-anchor the caller ("earlier you asked about SKU 8842, that's $34") or drop it into the middle of an unrelated sentence?
- user_updates_input_during_call — Caller corrects the input parameter mid-flight. Agent must re-issue the tool call and discard the stale result when it arrives.
- tool_timeout_graceful_fallback — Inject a tool that takes longer than your policy allows. Agent should surface an honest state to the caller and either escalate or offer a callback, never fabricate.
- barge_in_asks_for_status — Caller interrupts with "is that ready yet?" while pending. Agent should say still working; must not cancel, must not confabulate.
- silent_completion_check — Result returns cleanly. Agent must actually speak it, in a bounded time budget, without needing a caller nudge.
- parallel_calls_ordering — Two async tools in flight. Results can return in any order. Agent must attribute each to the right topic.
You want each of these run across a few personas: a rushed caller, a patient caller, a caller with a strong accent, one on a bad line. The failure modes surface differently across them.
How to test them, honestly
You cannot test any of this against a text harness. The failure modes live in timing, in audio, and in the coupling between what the model wrote and what it actually said. Three requirements fall out of that.
Real telephony, real streaming. The audio path, the endpointer, and the barge-in threshold all interact with the async-call timeline. A text-loopback harness will not reproduce the moment the caller interrupts a half-second into the agent's acknowledgment. You need the model running over the same transport it will run on in production: PSTN for phone agents, WebRTC where that applies.
Injectable tool timing. Your fixtures for lookup_price, check_inventory, schedule_dispatch need to be controllable. Force a 500ms return in one run, an eight-second return in the next, a timeout in the third, an out-of-order return in the fourth. Async failures are timing failures; if your tool fixtures always return in 200ms, you'll never see them.
Audio-native scoring. Because the model can write one value and speak another, and because premature-commitment failures often read fine on the transcript, you need to score the audio. Time-align the agent's utterances against the tool call ledger. Flag any numeric or identifier utterance that fires before its underlying tool has returned. Flag any spoken value that disagrees with the value the model wrote to the transcript.
Where Roark fits
This is what Roark is built for. Roark's simulation testing dials your voice agent over real phone calls (PSTN or WebRTC), not a text loopback, so the async timeline your model actually experiences is the one that gets tested. Simulations are built from personas, scenarios, and run plans, and the scenarios can be scheduled to gate deploys over HTTP.
Every call, simulated or production, is scored against 64+ built-in metrics plus any custom metrics you define, and the scoring is audio-native: the models grade the sound of the call, including pace, pauses, and interruptions, not just the transcript. That is the layer where premature-commitment and audio-text divergence become visible. When a call fails a check, Roark files an issue automatically, and you can turn the failing production call into a replayable regression test the next time you swap models, which is exactly the moment async behavior tends to shift. If you're wiring this into an agent built on Vapi, Retell, LiveKit, Pipecat, Bland, or ElevenLabs, the integrations are one-click; see the docs for the SDK details.
Don't wait for the next model swap to find out
The last time this landscape shifted, teams found out their agents behaved differently in production after they deployed. Async tool calls are a bigger jump than a routine version bump, because they change what "the agent is talking" and "the tool is running" mean in the same wall-clock second. Community discussion of the Realtime SDK captures the migration cost honestly: with the older synchronous implementation, execution paused until the function completed, which caused long periods of silence if the function took time to run. Async fixes that specific problem. It introduces the ones above.
The right response is a test suite for the async patterns before you turn them on, run over real calls, scored on audio. Ship the concurrency, then trust it because you tested it.
