Your voice agent finished a clean three-minute call. The caller sounded happy. The transcript reads well. Then the summary that hit your CRM has the wrong appointment date, the structured extraction dropped the last digit of the phone number, and the webhook fired to your scheduling system with an insurance member ID that never existed. Nobody on the call would have noticed. Nobody reviewing the transcript would either.
This is the part of a voice agent stack that most teams do not test. The call itself gets QA attention, the prompt gets iterated, latency gets tuned. The post-call write-back layer, the summarization, the structured extraction, the webhook payload, the human-handoff note, quietly ships whatever the last model returned. It is where the caller experience ends and the operational damage begins.
The write-back layer just got a lot more expensive to get wrong
Voice AI is moving from "the agent answers the phone" to "the agent runs the workflow." <cite index="15-1">Assort Health has developed voice AI agents that can handle the entire patient journey, inbound calls, triage, lab requests, med refills, scheduling, insurance eligibility and intake in any language.</cite> That platform just raised $120M at a $1.2B valuation, and <cite index="15-10">the platform runs the operational work behind each visit and writes every detail back to the electronic health record, including referrals, document processing, patient intake and personalized pre-post visit forms</cite>. The pitch is not the conversation. The pitch is the write-back.
The same shift is happening across the ecosystem. Vapi's structured outputs feature lets you define a JSON schema and get validated fields back at end of call, and <cite index="25-5,25-6">extracted data is included in call completion webhooks and available through the REST API, so existing application infrastructure receives clean, typed data without additional parsing layers</cite>. Retell's post-call analysis does the same thing with Boolean, text, number, and enum extractions, available via dashboard, webhook, or API. These outputs are what your operations actually consume. They are also what nobody is testing.
What the write-back layer actually contains
When someone says "the call ended," the platform is really firing off a chain of artifacts:
- A free-text summary. In Vapi this lands in
call.analysis.summaryat end of call. It gets pasted into CRM notes and read by whoever handles the follow-up. - A structured extraction. A JSON object matching a schema you defined:
appointment_time,patient_dob,insurance_member_id,refill_requested,escalate_to_human. This is what your systems act on programmatically. - A success evaluation. A pass/fail or rubric-score determined by an LLM looking at the whole transcript. This is what your dashboards aggregate into "resolution rate."
- An end-of-call webhook payload. The End of Call Report delivers all of the above to your server URL, usually within a few seconds of hangup.
- A handoff note. On warm transfer, whatever context you paint for the human agent, generated by the same summarization pipeline.
Each of these is produced by a different model call, on a different prompt, sometimes running on a different provider. Every one of them can be wrong while the conversation itself sounded perfect.
Why the write-back layer fails silently
Live-call QA does not catch this class of failure. Here is why.
The extraction runs on a prompt nobody is looking at. Structured outputs typically use a second LLM pass after the call ends. <cite index="24-3">After the call ends, structured outputs analyze the full transcript, messages, tool call results, and call metadata</cite>. That means the summarization prompt is a separate artifact from the agent prompt. When someone tweaks the agent to be more concise, the summary prompt still expects the old behavior and starts producing garbage.
HIPAA mode hides the evidence. Vapi's docs note that <cite index="28-1,28-2">when HIPAA mode is enabled, Vapi does not store structured outputs by default, which protects privacy but limits your ability to view structured outputs in Insights and Call Logs</cite>. That is exactly right for compliance, and exactly wrong for QA. Now the only place your extraction exists is in the webhook payload your own server saw. If you did not log it, it is gone. Teams building HIPAA agents routinely discover that they must validate extraction logic server-side because the platform will not show it to them.
Mishears compound. If the ASR heard "March twelfth" as "March second" early in the call, and the agent said "Great, so March second" and the caller mumbled "yeah," the transcript now contains the wrong date twice. The extraction prompt has no reason to flag it. The CRM gets a confidently wrong appointment. The caller shows up on the twelfth to a full room.
Model swaps break extraction quietly. Vapi normalizes structured outputs across providers, but <cite index="25-3,25-4">structured outputs work across OpenAI, Anthropic, Gemini, and other providers, and Vapi handles provider-specific implementations and normalize responses, so changing LLM providers doesn't break your data extraction logic</cite>. Normalized does not mean identical. A model swap changes edge-case behavior on ambiguous transcripts, and if you did not regression-test the extraction against a fixed set of calls, you will not see it until Ops complains.
Extraction schemas drift out of sync with downstream systems. Someone adds preferred_pharmacy to the JSON schema. Nobody adds it to the CRM mapping. Six months later a manager asks why the pharmacy field is never populated.
The QA plan for the write-back layer
Treat the write-back the same way you treat the call: define what "correct" means, test it at scale before launch, and score every production instance against the definition afterwards.
1. Define the write-back contract per intent
For each intent your agent handles (appointment scheduling, refill request, insurance eligibility, callback), write down the exact fields the downstream system expects and the exact meaning of each. Not the JSON Schema, the semantics. What counts as a valid appointment_time? Is "next Tuesday" acceptable, or must it be an ISO timestamp resolved against a timezone? What is the fallback when a field is not confidently extractable, null, empty string, or a needs_review flag? Get the answers written down before you write test cases.
2. Test the extraction against known ground truth, not against the transcript
The most common testing mistake is running the extraction prompt against a hand-picked transcript and eyeballing the JSON. That tests the extractor against itself. What you actually need is calls where you know the ground truth: the persona said their DOB was March 12, 1974. The extracted patient_dob field must equal 1974-03-12. Score it.
3. Test the summary as a metric, not a vibe
"The summary reads fine" is not a test. Pick the two or three claims that matter most for downstream action, appointment time, decision made, follow-up requested, and score whether each appears correctly in the summary text. LLM-as-judge is fine here, but the judge needs the ground truth from the scenario, not just the transcript, or it will grade the summary against the same misheard input the summary was built from.
4. Regression-test on model or prompt changes
Any change to the underlying model, the summarization prompt, the extraction schema, or the agent prompt should re-run the write-back suite. The moment "we upgraded to the new model tier" ships without a write-back regression, you have shipped a silent data-quality change.
5. Replay real production calls against updated logic
Every field mismatch that a customer notices in production is a test case waiting to happen. Capture it, freeze the audio, and replay it against your updated pipeline. If the fix works, the replay passes. If a future change breaks it, the replay fails.
What this looks like in practice
For a healthcare agent doing intake and scheduling, a real write-back test suite has scenarios like:
- Caller with a hyphenated last name spells it once, agent confirms. Assert extracted
last_namematches exactly, hyphen included. - Caller gives DOB as "March twelfth, seventy-four." Assert extracted
patient_dobis1974-03-12. - Caller reschedules mid-call from Tuesday to Thursday. Assert
appointment_timereflects the final agreed slot, not the first one mentioned. - Caller says "I don't have my insurance card, I'll call back." Assert
escalate_to_humanis true andinsurance_member_idis null, not a hallucinated string. - Background noise on the caller side, agent asks for repeat, caller repeats. Assert extraction succeeds despite the retry pattern in the transcript.
- Caller code-switches between English and Spanish. Assert the summary reflects the actual intent, not one language's version of it.
None of these are edge cases. Each is a class of call your agent will see hundreds of times a week.
Where Roark fits
The QA plan above is what you should do regardless of tooling. Building it yourself means a scenario runner, a telephony harness, a scoring layer that understands both audio and structured JSON, and a way to correlate the live call to the webhook payload your server received. That is a real engineering commitment.
Roark is built for exactly this shape of problem. Simulation testing dials your agent over real phone calls, PSTN and WebRTC, with personas that carry known ground truth: names, dates, insurance IDs, preferred pharmacies. When the persona says "my date of birth is March twelfth, nineteen seventy-four," the scenario carries that value, so any downstream extraction can be scored against it, not against the transcript that may already contain a mishear. Roark scores calls on 64+ built-in metrics plus unlimited custom metrics, so the write-back contract you defined in Step 1 becomes a custom metric that runs against every simulated and every live call.
The audio-native side matters here more than in most QA workflows. A summary can be exactly correct against a misheard transcript and still be exactly wrong against the actual audio. Roark's audio-native models score the sound of the call, so a mispronounced date or a misheard digit gets flagged at the call level even when the transcript looks clean, and the write-back mismatch downstream can be traced back to the original acoustic event via OpenTelemetry traces.
Production replay closes the loop. When Ops finds a real call where the CRM got the wrong appointment date, that call becomes a regression test. Any future change, new model, tweaked summarization prompt, updated extraction schema, re-runs against it. For healthcare deployments where a wrong write-back is a compliance and safety event, that regression suite is the difference between "we caught it in staging" and "we caught it in the EHR."
The call ends when the write is correct
The framing every voice-AI team should internalize: a call is not complete when the caller hangs up. It is complete when the downstream systems have the right data. Every intermediate artifact between those two points, the transcript, the summary, the structured extraction, the webhook payload, the handoff note, is a place where the agent can pass and the system can still fail.
Test all of it. Score all of it. Ship the write-back with the same rigor you ship the conversation, or accept that the number your CFO sees for "successful calls" is going to diverge from the number your Ops team sees for "correctly written records," and the gap is going to widen every time you swap a model.
