Testing appointment scheduling in voice AI agents

A missed call is a lost lead. A wrongly-booked appointment is worse: the caller hangs up believing a Tuesday 2pm slot is theirs, the calendar shows Wednesday 2am, and nobody finds out until someone doesn't show up. Scheduling agents are the most-shipped voice AI use case for a reason. They are also the ones where a silent failure looks, on the transcript, like a successful call.

Fall-back happens in most of the US on November 1, and there is a public write-up this year of a voice agent built on ElevenLabs that was calling a Cal.com check_availability tool with a UTC window instead of the caller's local time. The developer only found it because Claude Code read the raw transcript and spotted the timezone conversion that never happened. That is a delightful debugging story and a terrifying QA story: the agent sounded fine, the tool call succeeded, and the appointment was wrong.

This is the QA playbook we hand to teams shipping scheduling agents on Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs. It focuses on the failures that survive prompt tuning and show up in production as booking errors, double-bookings, and phantom no-shows.

Why scheduling agents fail in ways scheduling websites don't

A web booking form has a dropdown of valid slots. A voice agent has to interpret "next Tuesday, ideally in the morning, but not too early because I drop my kid off" and turn it into an ISO timestamp inside a specific timezone, with the right visit type, at a provider who is actually working that day. Every one of those steps is a place to fail.

The scheduling stack for a modern voice agent typically involves the LLM, a tool call to a calendar or EHR (Cal.com, Google Calendar, Athenahealth, Epic, HighLevel), and confirmation back through TTS. HighLevel's own guide is explicit that AI working hours and calendar availability are separate settings that both have to be right, and that misconfigured booking rules are the most common source of "booking errors and confusion when Voice AI begins offering live appointment times."

The failure modes we see repeatedly:

• Relative-time parsing off by a day. "Next Friday" spoken on a Thursday. "This weekend" spoken on Sunday afternoon. "Tomorrow morning" when the call rolls past midnight.
• Timezone drift. Caller is on the East Coast, agent runs in UTC, calendar is stored in America/Chicago. Any single link in that chain that assumes its own local time silently books the wrong slot.
• DST edges. The hour that doesn't exist in spring, and the hour that happens twice in fall. If your tool call constructs a datetime by string concatenation, one of them will bite you.
• Availability desync. The tool returned open slots at the start of the call. Sixty seconds of conversation later, another caller took one. The agent still offers it.
• Wrong appointment type. Bland's team makes the billing consequence explicit: a new-patient visit booked as an established one is a coding mismatch that appears in claims weeks later.
• Provider or location leakage. Agent offers a slot at a location the caller didn't ask for, or a provider they can't see under their insurance.
• Confirmation that doesn't match the write-back. Agent says "Tuesday at 2," and the ISO timestamp posted to the calendar is Wednesday at 2am. The caller hears the right thing. The system of record disagrees.

None of these show up as unhappy calls. The audio is polite. The tool call returns 200. The write-back looks like every other write-back. You need tests that check what actually got booked, not what the agent said.

Build the test matrix from the caller's phrasing, not the calendar's fields

The temptation is to enumerate every calendar field (provider, location, appointment type, duration) and write a test per combination. That surfaces some bugs but misses the interesting ones, because callers don't speak in fields. They speak in phrases whose interpretation is exactly what you are testing.

Start from the utterance side. For a healthcare booking agent, we build the matrix along five axes:

1. Time phrasing. Absolute ("March 4th at 3pm"), relative ("next Wednesday", "the Tuesday after next"), fuzzy ("sometime next week", "any morning is fine"), constrained ("after work but before 6"), and multi-turn ("actually, can we do the following day instead").
2. Timezone context. Caller confirms zone, caller does not confirm zone, caller travels ("I'm in Denver but my appointment is in the New York office"), caller is on the DST boundary date.
3. Availability shape. Slot is open, slot is taken, slot is on a blocked day, provider is on vacation, only one option left, nothing that week.
4. Booking type. New patient, established, telehealth vs in-person, follow-up requiring specific provider, multi-service.
5. Change intent. First booking, reschedule with the same duration, reschedule shorter, cancel with a waitlist, cancel outright.

Cross those and you get a scenario grid in the low hundreds, which is where the interesting bugs actually live. The Retell scheduler docs frame the same idea from the platform side: "simulate conversations and edge cases to validate fallbacks, ambiguous inputs, accents, interruptions, and scheduling conflicts." The word doing the work in that sentence is simulate.

Simulate before you ship, on the actual telephony path

Text-based prompt evaluation catches a fraction of these. The rest live in ASR, endpointing, and the exact JSON the LLM emits to the tool. Those you can only see over the same call path a real caller takes.

This is where Roark starts. Roark's simulation testing dials your agent over real phone calls (PSTN) and WebRTC, driven by a persona and scenario library you control. Personas set the voice, accent, speech pace, background environment, and emotional register; scenarios encode what the caller wants, what constraints they carry, and what "correct" looks like at the end. For scheduling, the correct end-state is a specific ISO timestamp, a specific visit type, and a specific provider, written to a specific calendar. If the agent's transcript agrees but the write-back disagrees, the scenario fails.

Two capabilities matter specifically for scheduling QA:

• Automated runs on a schedule. Recurring simulation runs before each DST transition, before each release, and on a nightly cadence catch the class of bug where nothing about your code changed but the world did. The week of November 1 is exactly the right week to have a suite running against every "next Sunday morning" phrasing your customers use.
• Production call replay. When a live call goes wrong, capture it and replay the same audio against the fixed agent. That is how a scheduling bug becomes a regression test instead of a war story. Roark supports this natively: real production audio, replayed against updated agent logic.

Score the live calls too, not just the simulations

Simulation catches the classes of failure you thought of. Production catches the ones you didn't. Every scheduling call in production is worth scoring, and the metrics that matter for scheduling are audio-native and slot-level, not just "did the LLM sound helpful."

The scoring rubric we recommend for a scheduling agent, in rough priority order:

Roark ships 64+ built-in metrics plus unlimited custom ones, and the audio-native models score pronunciation, pace, pauses, and interruptions in addition to what the transcript says. Every production call is scored, and when a metric fails, an issue is filed automatically so the failed calls don't get lost in a review queue nobody reads. For regulated verticals, the healthcare industry page covers the compliance posture (SOC 2 Type II, HIPAA BAA on request).

The pattern to avoid is sample-based QA on scheduling calls specifically. If your booking success rate is 95%, and you review 50 calls a week, you will read 47 successful bookings, two escalations, and one call where the agent handled a rescheduling gracefully. The wrong-timezone booking sits in the 95% until a patient calls back angry three weeks later.

A ship-checklist for scheduling agents

Before a scheduling agent goes live, or when a model or prompt changes, we hand teams this checklist. Every item is a scenario category, not a single test.

1. Relative-time coverage. "Today", "tomorrow", "this weekend", "next week", "the week after next", each said at four times of day, including one that crosses midnight during the call.
2. Timezone coverage. Caller in one zone, agent in another, calendar in a third. Include Arizona, which does not observe DST, and one caller who names their zone incorrectly.
3. DST boundary. Book slots that fall inside the fall-back hour that happens twice, and inside the spring-forward hour that does not exist. Do this against the actual DST-transition date, not just "any Sunday."
4. Availability race. Two simultaneous simulated calls, both trying to grab the last slot of the day.
5. Wrong-type traps. New-patient callers who describe themselves as returning, telehealth requesters who default to in-person.
6. Reschedule and cancel. From the same phone number as the original booking, from a different number, with and without the confirmation code.
7. No-slot fallback. Nothing available in the caller's window. Does the agent offer waitlist, offer the next opening, or invent a slot?
8. Confirmation readback. Full date, day of week, time, provider, location. Agent must say all five before hanging up.
9. After-hours behaviour. Calls that arrive when the agent's working hours and the calendar's working hours disagree.
10. Regression suite from every real failure. Every production call that failed a metric becomes a replay scenario that must keep passing.

Run the checklist as automated simulations, not as manual test calls. A human running through this list once takes a day. A simulation suite runs it every night and every deploy.

Where this connects

The scheduling agent is where a lot of teams learn that voice AI has a different QA shape than the rest of their stack. Prompts drift, models tick a version, DST arrives, and the same agent that shipped fine last quarter starts booking Wednesday 2am appointments. The way out is the boring one: real telephony simulation on a scenario library that covers the phrasings your callers actually use, audio-native scoring on every live call, and a replay loop that turns each real failure into a permanent test.

If you are running a scheduling agent on Vapi, Retell, LiveKit, Pipecat, Bland, or ElevenLabs, Roark plugs in as a one-click integration and the SDK docs cover the ingestion path if you are wired up more directly. The calls you are already recording become the regression suite you never had time to build.