On September 10, 2026, Intermedia launched AI Receptionist, a native AI voice agent for its Unite and Contact Center platform. The interesting detail is not that it exists. It is how it gets configured: through an AI Builder that can pull in FAQs, business details, and knowledge collections built from uploaded documents and scraped web content. ElevenLabs Agents works the same way. So does Retell, which will re-fetch URL sources every 24 hours if you turn on auto-refresh.
Which means your Ops manager updating the intake PDF at 3pm, your CX lead editing the returns policy on the help center, and your marketing team publishing a new pricing page are all deploying to a production voice agent. Silently. Without a CI job, a code review, or a QA pass. This post is about building a testing process that catches that.
Why a knowledge base edit acts like a code deploy
Voice agents grounded on your documents follow a simple loop: the caller asks something, the retriever fetches the most relevant chunks from your knowledge base, and the LLM generates a spoken answer from those chunks. Change the source, change what gets retrieved, change the answer. As one recent CX Today piece put it, if the knowledge base is outdated, RAG just retrieves the wrong answer faster.
The problem is that this deploy pipeline has no gates. Product code goes through PRs, staging, and smoke tests. A Google Doc edit propagates the moment someone hits save. When ElevenLabs' knowledge base or Retell's auto-refresh re-indexes a scraped URL, nobody gets paged. The next caller is the smoke test.
For text chat, the blast radius is small. A wrong sentence gets read on screen, the customer copy-pastes it into a support ticket, someone fixes it. Voice is different. The caller cannot scroll back. The wrong price, the missing disclaimer, the fabricated policy: they land as spoken statements the caller acts on. And on a phone, tone carries as much weight as content. An agent that confidently states a wrong hours change sounds exactly like an agent that confidently states the right one.
The failure modes that ship with a KB edit
Five patterns show up over and over when a knowledge base changes.
Stale-chunk survival. The old content was cached, the new content was indexed, and the retriever still favors the old chunk because it happens to match the caller's phrasing better. The agent answers with last quarter's pricing and sounds confident doing it.
Adjacent-chunk drift. The doc got restructured. A section that used to say "we refund within 30 days" now sits next to a paragraph about a 90-day promotional window. Retrieval pulls the wrong neighbor. This is the failure that Zoice describes as "confident fabrication": the agent invents a coverage limit or return window that does not exist, in the same tone it uses for real facts.
Refusal miscalibration. Retell's guide to structuring voice agent knowledge bases recommends a hard refusal path when the top retrieved chunk falls below a similarity threshold. Good practice, but the threshold is tuned to your corpus. Change the corpus and the calibration drifts: the agent starts refusing questions it used to answer, or answering questions it used to escalate.
Dropped disclaimer. Someone tidied up the FAQ and cut the "this is not medical advice" line, or the "results not guaranteed" paragraph, or the recording-consent boilerplate. The retrieval still returns the surrounding content. The agent stops saying the disclaimer.
Voice-only regressions. This is the class most text-QA processes miss entirely. A new product name gets added to the docs: does the TTS pronounce it correctly? A new SKU gets listed: does the agent read the alphanumeric code back digit by digit? A new price tier appears: does the number get spoken as "one twenty-nine ninety-nine" or "one hundred twenty nine dollars and ninety nine cents", and either way, does the caller understand it? These failures do not appear in a text-based retrieval eval because the output was correct on the page.
Build a regression suite that runs on every KB change
Treat the knowledge base like source code. Every edit ships. Every ship gets a test suite. The suite has four ingredients.
Seed from real caller questions. Not what you think callers ask, what they actually ask. Pull the top 50 to 100 real questions from your call transcripts, cluster them, and turn each cluster into a scenario. This is the foundation Retell's guide recommends for its own retrieval tests: run real caller questions through retrieval and inspect what comes back before any LLM generation happens. Do the same at the audio level.
Include out-of-scope tests. Half your suite should ask questions the agent should not answer. If the refusal calibration slips, this is what catches it. A KB change that suddenly returns confident answers to "what's the CEO's cell number" is a failure the happy-path suite will never surface.
Include voice-specific checks. For every new proper noun, price, or alphanumeric identifier that landed in the KB update, add a scenario that forces the agent to speak it out loud. Score for pronunciation, pace, and readback correctness, not just semantic match.
Vary the persona. As Gladia has argued, noisy environments, accents, and language switching all create misunderstandings that spiral through the rest of the conversation. A KB regression that passes with a clean American English speaker in a quiet room can fail with a Scottish caller on a Bluetooth headset in a moving car. Both are your customers.
Then wire the trigger. When your CMS publishes, when your help center webhook fires, when your document store updates, kick off the suite. Do not wait for the next release cycle. The whole point is that content changes are not on a release cycle.
Score every live call after the change lands
A regression suite catches what you thought to test. It cannot cover the long tail: the caller who asks about the new product using the old product's name, the caller who quotes back a fragment of a page you did not know had been indexed, the caller who trips a code-switch between English and Spanish on a term that was added in one language only.
Continuous scoring on live calls is what closes that gap. Every production call gets evaluated against the metric suite. Anything that looks like a KB-related failure (contradiction with source, refusal drift, disclaimer omission, mispronunciation of a term you just added) gets filed automatically. The next morning, the KB owner sees the issues clustered by the change that caused them.
The valuable byproduct is that these real failures become future regression tests. The exact call that broke gets replayed against the fixed agent to confirm the fix, then stays in the suite forever. This is the loop Roark is built around: simulation testing over real phone calls before launch, audio-native scoring on every live call, and production call replay as regression coverage.
A launch checklist for knowledge base changes
Give content owners a checklist that translates the type of edit into the QA it requires. The exact thresholds are yours to set. The categories are not:
| Change type | Test to run |
|---|---|
| Pricing update | Price readback scenarios, disclaimer presence, currency handling for international callers |
| New product or SKU added | Pronunciation of the new name, alphanumeric readback of the SKU, refusal test for adjacent products that don't exist |
| Policy or T&Cs revision | Compliance disclaimer presence, direct-quote scenarios, out-of-scope refusal |
| Hours or availability change | Timezone-aware scenarios, holiday handling, escalation-to-human path |
| Help center page added | Retrieval smoke test that the new page is reachable at all, plus adjacent-chunk drift check |
| Help center page removed | Confirm the agent no longer answers with removed content, and refuses cleanly instead |
| Bulk re-index or auto-refresh | Full regression suite, because you don't know which chunks moved |
The checklist is not a QA gate that blocks the CX team from editing docs. It is a set of automated suites the platform runs on their behalf when they save. The goal is to make good testing invisible to the content owner and impossible to skip.
How Roark fits
Roark is a simulation-testing, observability, and reporting platform for production voice AI agents, and knowledge-base drift is exactly the failure mode the platform is designed to catch.
Simulations dial your agent over real telephony (PSTN and WebRTC), using personas that carry distinct voices, languages and accents, speech pace, emotional register, and background-noise environments. You build a KB regression suite as scenarios once, then trigger it from your content management system's webhook via HTTP so the run gates the change. Roark simulates and scores in 45 languages and accents, which matters when your knowledge base serves callers who don't all sound the same.
Every live call after the change gets scored against 64+ built-in audio-native metrics (plus custom ones for the checks specific to your product) covering pronunciation, emotion, vocal stress, pace and pauses, and interruptions. What breaks is filed as an issue automatically. Real production failures can be captured and replayed against updated agent logic as regression tests, so the same caller that broke you once cannot break you twice.
One-click integrations are available for Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs, and Roark is SOC 2 Type II certified with a HIPAA BAA available for regulated deployments.
The larger point
The moment your voice agent's answers come from documents someone can edit without a PR, you have a new class of deploy: the silent kind, done by people who don't think of themselves as deployers. Every FAQ edit, every help-center refresh, every scraped-URL re-crawl is a change to what your agent tells the caller on the phone tonight.
Treat those changes with the same discipline as code. Suite them, gate them, score the live calls that hit production after them, replay the failures. The alternative is finding out from a caller, which is the most expensive way to run QA that exists.
