All field notes

Evaluation Strategy

·

Platform certification is not launch readiness for voice AI agents

Microsoft and Twilio both pushed voice AI platform readiness this week. Here's why partner certification and infrastructure GA don't prove your agent is launch-ready.

James Zammit

James Zammit

Co-founder & CEO @ Roark

8 min read
Platform certification is not launch readiness for voice AI agents

Two readiness announcements landed the same week. At Ignite, Microsoft said voice agents in Foundry Agent Service are now generally available, connecting to Teams Phone, Twilio, and Genesys, and support is limited to a short list of certified partners. Two days earlier, Twilio pushed Smart Turn Detection, HD Audio, and automatic STT failover to GA inside Conversation Relay, with on-transcript hallucination and task-completion checks in public beta.

If you lead a voice AI program, both of these are useful. Neither of them, together or separately, tells you that your own agent is safe to turn on. That gap, between platform-level readiness and agent-level readiness, is where most voice launches quietly go wrong. Procurement teams tick certification boxes, engineering teams ship the integration, and the first hundred production calls discover what a demo never would.

What platform certification actually proves

Platform certification is a procurement artifact, and a legitimately useful one. Microsoft's Teams Unify certification for voice agents verifies that a participating provider's solution meets the quality, compatibility, and reliability Microsoft expects of its own products, and Microsoft will not support voice agent deployments from non-certified vendors. NiCE's Cognigy was among the first to complete certification for answering inbound Teams Phone calls and handing callers to the right person, queue, or department with context.

That is a real guarantee. Certification tells you the integration layer works: the call signaling is clean, the handoff into a human queue preserves context, the identity model plays nicely with your tenant, the vendor has an operational bar Microsoft trusts. Every one of those matters, because if any of them break, every conversation breaks.

The same reasoning applies to Twilio's Conversation Relay news. Smart Turn Detection, per Twilio's own changelog, distinguishes between a caller who has finished speaking and one who is pausing mid-sentence, across all supported STT providers (Deepgram Flux uses its own turn detection instead). HD Audio widens the frequency range passed between callers and the model. STT Failover switches providers during outages or latency spikes with no customer code required. Those features move the baseline of what a competent voice stack does. If you were previously burning engineering cycles writing your own semantic end-of-turn heuristics, you no longer have to.

What it doesn't prove

Here is what platform certification and infrastructure GA do not tell you, and cannot tell you:

  • Whether your agent collects the correct intent before transferring to a human
  • Whether your prompt handles a caller who lies, hesitates, barges in, or circles back three times
  • Whether your tools return the right data, and whether the model trusts them when they do
  • Whether a PHI or PCI disclosure gets handled, redacted, and logged the way your compliance team told the auditor it would
  • Whether the agent closes the loop at the end of a call, or trails off into a silent hangup
  • Whether a model refresh from your TTS or STT vendor next Tuesday quietly changes any of the above

A certified integration means the call will reach your agent cleanly and leave it cleanly. It does not care what happens in the middle, which is where almost all of the risk lives. The Microsoft certification page is explicit about this: support boundaries end at the partner's door, and incidents go to the voice agent partner before they ever reach Microsoft. Platform certification is a procurement statement about the vendor. Launch readiness is an operational statement about your deployment of that vendor. The two are not the same, and they never will be.

Platform certification vs. the readiness your own agent still owes you
Platform certification vs. the readiness your own agent still owes you

Three readiness signals that get confused

In practice I watch product leaders conflate three different readiness signals, and treat any one of them as if it stood in for the others.

Platform certification. The one we've been discussing. A vendor meets a hyperscaler's integration bar. It implies nothing about your specific agent, prompts, tools, or callers. It travels well in procurement decks because it is binary and verifiable.

Vendor compliance posture. SOC 2 Type II, HIPAA BAA, ISO 27001, PCI DSS. These answer the question "can I legally and contractually run a voice agent on this stack for this use case." They do not answer "will my agent behave correctly during a PCI-adjacent moment on a real call." As one practitioner guide puts it, compliance certifications don't prove an agent behaved within authorized scope during a live call.

Agent behavioral readiness. How your specific agent, with your specific configuration, handles the full distribution of calls it will actually receive. This is the only signal that correlates to launch outcomes, and the only one that cannot be bought from a vendor's website.

Procurement rewards the first two because they are easy to verify. The third is the only one that keeps you out of the newspaper after launch. A launch-readiness review that does not separate these three is not a review, it is a hope.

Build an agent-level readiness scorecard

If you are the person signing off on launch, the discipline worth imposing is simple: for every item your vendor's certification already covers, cross it off. For everything left, you own the evidence. In a typical deployment the remaining list looks like this:

AreaPlatform certification coversYou still owe evidence for
Call signaling and handoffYesContext preservation on transfer to your specific queue
Turn detection algorithmYes (where the platform ships it)Behavior on your real callers, accents, and hesitation patterns
STT/TTS failoverYes (where the platform ships it)Your agent's behavior during the switch, not just continuity
Prompt and tool behaviorNoEvery intent your agent is supposed to handle
Disclosure, consent, compliance phrasingNoActual utterances against actual regulated flows
Interruption and barge-in handlingPartialYour agent's recovery after a mid-sentence caller
End-of-call wrap-upNoEvery exit path, including the silent ones
Model refresh behaviorNoAny week your STT, TTS, or LLM vendor ships an update

The right-hand column is the agent-level readiness scorecard. It is not optional. It is the minimum bar for turning the system on, and it is the ongoing bar for keeping it on. A launch review that only produces the left column is a launch review that defers risk until the first thousand production calls, and then calls it bad luck.

Simulation is how you fill the right column

You cannot fill the right column by listening to calls. The volume is wrong, the sampling is wrong, and the humans doing the listening are too close to the agent they built. Industry practice in regulated contact centers is to review 2 to 5 percent of calls manually, which is to say, most compliance risk is undetected until an audit or a customer complaint surfaces it. That is not a QA process. That is a lagging indicator.

The only way to generate agent-level readiness evidence at the scale certification programs imply is to simulate the agent over real calls, before launch, against the scenarios you actually care about, and to score every live call the same way afterward. This is what Roark is built to do. We dial your agent over real phone calls, over PSTN or WebRTC, with personas that carry distinct voices, accents, speech pace, and background environments, and we score the audio directly, not just the transcript. The personas and scenarios are yours: every intent, every disclosure, every edge case your vendor's certification did not test against. Suites run on a schedule, and they can gate CI over HTTP so a prompt change that regresses a required disclosure never reaches production.

An illustrative pre-launch suite covering the behaviors a platform certification doesn't
An illustrative pre-launch suite covering the behaviors a platform certification doesn't

Every production call is then scored against the same metric suite you tested with. Roark ships 64+ built-in metrics covering pronunciation, emotion, vocal stress, pace and pauses, and interruptions, plus any custom metric you need to express a regulated flow or a product-specific rule. When a call fails, it files as an issue, and the audio and transcript are there to replay. The failed production calls themselves become regression tests, which is how you avoid relearning the same lesson every model refresh. For teams building on Vapi, Retell, LiveKit, Pipecat, Bland, or ElevenLabs, the integration is one click; the metric definitions and scoring runbooks are in the docs.

The uncomfortable part

Platform certification is attractive because it is a yes/no answer that someone else already decided. Agent-level readiness is a continuous answer that only you can decide. The temptation at any voice AI program is to let the easier signal substitute for the harder one, especially right after a vendor launches a shiny new certification badge.

Don't. The badge means your plumbing is clean. It does not mean your water is drinkable. For that you still have to test the water, keep testing it, and have the receipts when someone asks. If the week's news tells you anything, it is that the platforms are getting serious about the parts they own. The parts you own have not moved.

James Zammit

Written by

James Zammit · Co-founder & CEO @ Roark

Building Roark — the quality platform that simulates, monitors, and auto-improves voice and chat agents.

Bring a recording.
We’ll score it live.

See your own agent measured on the audio it actually produced, in the demo, in real time. Stop guessing whether your voice AI works.

Or get $50 in credit with a work email · read the docs · support@roark.ai