All field notes

Voice AI Testing

·

Testing voice agents for AI callers

Google's agent now dials businesses on behalf of homeowners. When your voice agent picks up, the caller is another AI. Here's what to test before it does.

Daniel Gauci Mizzi

Daniel Gauci Mizzi

Co-founder & CTO @ Roark

9 min read
Testing voice agents for AI callers

Google's AI now dials local businesses on behalf of homeowners. It asks the questions the user typed, wants a structured answer, and hangs up on shops that fumble. At I/O this year, Google confirmed that "for select categories like home repair, beauty or pet care, you can ask Google to call businesses on your behalf," and that these capabilities rolled out to everyone in the U.S. this summer. Consumer calling agents from Assindo, Siri, and a wave of well-funded voice startups are arriving at the same conclusion within a few weeks of each other.

If your voice agent answers the phone for a services business, the caller on the other end is increasingly not a person. It is another agent, working through a checklist, listening for specific tokens, and ready to hang up and try your competitor if the shape of the conversation deviates from what it expects. Every assumption your QA suite makes about the caller was built for humans. That's the problem.

The AI-to-AI phone call is not a thought experiment

Google's Business Profile documentation describes the feature as available "only in the United States, except in these states:" Indiana, Louisiana, Minnesota, Montana and Nebraska, with calls "monitored and recorded for quality assurance". An unverified business must verify before it can opt out. This is not a research demo, and it is not narrowly gated. It is a live consumer surface running against the general population of small business phone lines, and the businesses picking up are increasingly answering with their own voice agents. Two models negotiating a haircut over a phone line is now the default path for a real slice of local commerce.

The industry roundups from the last two weeks all read the same way: Google shipping a consumer agent that calls a store is the clearest signal yet that agents are moving from describing things to doing them, and the funding data says the industry is committed. In parallel, enterprise-facing voice agents have crossed the "done piloting" threshold, with vendors at Customer Contact Week openly telling buyers that the test phase is over. Both sides of the call are automating at once.

Old vs. new caller assumptions for enterprise voice agents
Old vs. new caller assumptions for enterprise voice agents

What actually changes when the caller is a machine

Your QA scenarios were written for a human caller who hesitates, backchannels, gets distracted, mishears, apologizes, and repeats themselves. A machine caller does none of that. The behaviors shift in ways that matter for testing:

  • No filler, no hesitation. Machine callers speak in dense, complete sentences. Endpointing tuned for the natural gaps in human speech may not fire when expected, or may cut off mid-utterance because the caller's speech has no prosodic downturn.
  • Structured, multi-part requests. The prompts these agents are trained on look nothing like "Yeah, hi, so I wanted to ask about... um, do you do water heaters?" One published example of a Google agent's opening turn asks for availability, installed price including permit, warranty terms, haul-away, and recent reviews under 4.5 stars, all in one go. Your agent's one-thing-at-a-time flow is not designed for that payload.
  • Hard latency budget. The same source describes the machine caller as getting a structured quote in under 90 seconds or hanging up and trying the next shop. Human callers tolerate roughly 800 milliseconds of response latency before conversation quality degrades. Machine callers are stricter, and they measure total time to a usable answer, not per-turn latency.
  • No backchannels. Humans say "mm-hm" and "right" and "okay" during your agent's speech. Machine callers wait silently, which changes how your barge-in and voice activity detection behave in the presence of a live audio channel with almost no signal on the caller side.
  • Deterministic scripts. The caller will re-ask the exact same question if it doesn't get a machine-parseable answer, sometimes verbatim. Your agent's fallback and clarification paths need to converge, not loop.
  • AI disclosure loops. If your agent is required to disclose that it's AI, and the caller is another AI trained to disclose, the greeting can become a two-sided disclosure exchange that neither side is designed to end. This is a real failure mode in early recordings.

None of these are exotic. They are the mechanical consequence of one endpoint being deterministic software. But almost every voice-agent test suite in production today treats the caller as a distribution of humans, not a mixed population that now includes precise, patient, unforgiving machines.

The failure modes your suite probably misses

Once you look for them, the AI-to-AI failure modes cluster into a short list. In no particular order:

  1. Endpointing collapses. The caller's utterance runs 12 seconds without a prosodic pause. Your VAD keeps waiting. Your agent responds late, or worse, interrupts because the caller finally paused for breath.
  2. Multi-slot answer misses. The caller asked five things. Your agent answers the first, offers to help with more, and the caller hangs up because it didn't get slots two through five.
  3. Confirmation dance breaks. Your agent says "Just to confirm, that's a 50-gallon water heater, installed by Friday, right?" The machine caller answers "Yes" and then re-asks its original five questions. Your state machine thinks it's still confirming.
  4. Disclosure never resolves. Both sides read a scripted AI disclosure. Neither has a rule for what to do when the disclosure it just heard was also from an AI.
  5. Silent-caller misclassification. During your agent's speech, the caller is fully silent, no ambient noise, no breath. Some pipelines treat this as line drop and terminate the call.
  6. Voice-cloning-adjacent risk. Vectra AI's report found AI-powered scams surged 1,210% in 2025, with voice cloning identified as a top enterprise risk. Not all machine callers are legitimate. Your agent's authentication flows need to hold against a caller that sounds exactly like a known customer.

Each of these has a fix, and most of the fixes are small. The problem is they don't get found until a production call, because your test set doesn't contain a caller that sounds like an agent.

A machine-caller persona set

The most useful thing a QA lead can do this month is add a machine-caller cohort to the persona set. Not one persona, a cohort, because "AI caller" is not a single shape any more than "human caller" is. A workable starting cut:

PersonaBehaviorWhat it tests
The information agentAsks 3 to 7 structured questions in one turn, wants a machine-parseable answer, hangs up at 90 secondsMulti-slot answer coverage, latency budget, structured response
The booking agentConstraint-driven ("Friday after 3pm, under $400"), negotiates within a rangeConstraint handling, offer/counter-offer flow, calendar tool calls
The IVR navigatorDeterministic path through your menu, retries verbatim on mismatchFallback loops, timeout handling, path convergence
The polite agentDiscloses as AI in the greeting, waits for reciprocal disclosureDisclosure exchange doesn't loop, greeting completes
The fast talker1.4x normal speech pace, no filler, no breathEndpointing at high WPM, ASR error rate on dense speech
The impersonatorCloned voice matching a known customer's timbre, requesting account changesAuthentication flows, out-of-band verification, high-risk-action gating

The first five are QA problems, not security problems. The last one is a security problem, and it belongs in your test suite for exactly the same reason: it is a caller you will see in production, so your agent needs to have been tested against it before launch.

Machine-caller test suite results
Machine-caller test suite results

Turning real AI-to-AI calls into regression tests

The hardest part of testing for machine callers is that the calls sound legitimate right up until they fail. There's no dropped-audio glitch, no obvious accent mismatch, no baby crying in the background. The failure is silent: your agent gave a fine answer to the first question, and the caller quietly hung up.

The only reliable way to find these is to instrument production and pull the failed calls back into the test loop. Two rules make that concrete:

  1. Tag machine callers on ingest. Even a rough heuristic works to start: no ambient noise, no interruptions from the caller during your agent's speech, unusual regularity of pace. Tagging lets you slice metrics by caller type and see whether your P95 latency, hang-up rate, and slot-fill accuracy differ for machine callers. They almost certainly do.
  2. Replay failed machine-caller calls against the next agent build. A machine-caller call is uniquely well-suited to replay: the caller side is deterministic enough that a captured audio stream is a fair regression test on its own. When your agent updates, the same call should now pass, and the fix should not regress the human population.

This is the workflow that actually catches AI-caller regressions before they hit revenue: production sample, tagged as machine caller, replayed into simulation on every deploy, promoted to regression suite when a real failure gets fixed.

From a live AI-to-AI call to a regression test
From a live AI-to-AI call to a regression test

Where Roark fits

Roark was built for exactly this loop. The simulation layer dials your agent over real telephony (PSTN and WebRTC), not a text loopback, so the acoustic and timing behavior of a machine caller shows up the way it will in production. Personas are configurable along the axes that matter for AI callers: distinct voices, languages and accents, speech pace, emotional register, and background-noise environment. You can build a "fast, precise, no-filler" caller, a "constraint-driven booker," or a "polite discloser" and run them against your agent on a schedule, gated in CI.

Every production call gets scored against Roark's audio-native metrics, which look at the sound of the call and not only the transcript: pronunciation, pace and pauses, interruptions, vocal stress. Failures are filed as issues automatically. When a real machine-caller call breaks something, you can replay that captured production call against updated agent logic and pin it into the regression suite. Roark has one-click integrations for Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs, so this fits into whatever stack you're already running. For services businesses that Google's agent will actively be dialing, the industry-specific setup lives on the home services page.

A short checklist before your next release

  • Add a machine-caller persona cohort to your simulation suite. Six personas is plenty to start.
  • Verify your agent answers a five-slot question in a single response, not one slot at a time.
  • Confirm your greeting handles a caller that also opens with an AI disclosure.
  • Measure P95 time-to-usable-answer, not just per-turn latency, and set a target below 90 seconds for a services quote flow.
  • Tag production calls as machine callers on ingest, and slice all your dashboards by that tag.
  • Replay every failed machine-caller call against the next build before you promote.

The AI-to-AI phone call is not the exotic case any more. It is the case your suite is quietly missing on every deploy, and it is the one where losing a call means losing a customer to whichever competitor's agent picked up faster.

Daniel Gauci Mizzi

Written by

Daniel Gauci Mizzi · Co-founder & CTO @ Roark

Building Roark — the quality platform that simulates, monitors, and auto-improves voice and chat agents.

Bring a recording.
We’ll score it live.

See your own agent measured on the audio it actually produced, in the demo, in real time. Stop guessing whether your voice AI works.

Or start free with $50 in credit · read the docs · support@roark.ai