The last two weeks have been loud. Cartesia launched Managed Agents, a hosted voice-agent platform that claims the top spot on Artificial Analysis's Speech Agent Arena and on ServiceNow's EVA-Bench at launch. Convoso's no-code Voice AI Agent went generally available on November 2. OpenAI's GPT-Live can now dial outbound through any SIP trunk. And Regal's Copilot has been marketing self-improving agents that run their own tests and quietly tune themselves between calls.
The pitch in every one of these launches is the same: the platform will build the agent, deploy the agent, measure the agent, and in some cases improve the agent, all inside one product. If you are the PM holding the quality bar, the question that lands on your desk is the obvious one. Do I still need my own QA? The answer, every time, is yes, and this post is the argument you can hand your leadership team when they ask why.
The lifecycle claim is real. The QA claim is not.
Vendor-owned lifecycle platforms are doing something genuinely new. Regal's Copilot generates call scenarios, runs tests, and suggests improvements before the agent goes live, and keeps doing it after launch. Cartesia tuned listening, turn-taking, reasoning, tool-calling and speech together rather than bolting them into a pipeline. These are serious products built by serious teams.
But none of this replaces QA, because QA is not a step inside the build loop. QA is the independent check on the build loop. When the system that writes the agent is the same system that scores it, and the same system that decides what to fix, there is nowhere in that loop for your definition of a good call to enter.
That matters because your definition is not the vendor's. Your definition includes the escalation path your legal team signed off on, the one phrase the compliance officer refuses to let the agent say, the knowledge-base article that was updated last Tuesday, the fact that your caller base skews older in the Midwest, and the fact that revenue only clears when the agent completes the specific sequence of three tool calls your billing system expects. A platform-generic evaluator cannot see any of this, and a self-improving loop can only optimise against what it can see.
Three failure modes PMs should name out loud
1. Benchmark scores do not predict your traffic
The industry is drowning in benchmark numbers. Cartesia Managed Agents launched #1 of 30 agents on Speech Agent Arena with a preference Elo of 1115, and posts 87% accuracy and 91% experience on EVA-Bench across 200+ scenarios. Phonic-1 recently topped the Artificial Analysis speech-to-speech index, scoring 83.2 on τ³-Voice versus 79.9 for GPT-Live-1. These are real numbers. They are also someone else's scenarios.
EVA-Bench is a customer service, IT service, and HR benchmark. If you run a home-services dispatch agent that has to collect an address, confirm a time window, and verify a payment method in under ninety seconds, the Elo on an arena of general-purpose conversations tells you very little about whether the agent will complete your job. The number is a starting filter. It is not evidence of readiness.
2. Self-improvement optimises against vendor-defined labels
The appeal of a self-improving platform is that the agent gets better while you sleep. The cost, rarely spelled out in the launch post, is that it gets better against whatever labels the platform uses to decide what "better" means.
If the platform defines a successful call as "no hangup, no explicit negative sentiment, tool call returned 200," the self-improvement loop will drive the agent toward calls that pass that filter. Those are not the same as calls your business considers successful. A caller who hangs up politely after the agent subtly discouraged them from escalating counts as a success in the first frame and a lost conversion in yours. A tool call that returned 200 but wrote the wrong appointment time counts as a success in the first frame and a next-week complaint in yours.
Self-improvement is a powerful mechanism. It is also a lossy one when the loss function is not under your control.
3. The vendor cannot test the integration you own
Even a perfectly scored vendor agent runs inside your stack. It hits your CRM, your scheduling system, your knowledge base, your fallback IVR, your fraud rules. The vendor cannot see any of that during their internal evaluation. They test their agent. You ship a system.
Launches like Soniox's STT v6, which can take the agent's text over the same WebSocket and use it as context while removing the agent's own voice from the transcript, make the integration surface richer, not simpler. More capability at the edges means more things to verify end to end before launch.
What independent voice QA actually requires
Independent QA for a voice agent is not "run some prompts through the vendor's eval UI." It has three load-bearing properties.
It uses your scenarios, your personas, your traffic. A good test suite reads like your call recordings. The caller interrupts, has an accent the agent has struggled with, calls from a noisy environment, asks the two questions your agent gets wrong most often. These scenarios are not in any public benchmark and will never be in a vendor's default evaluation set. They are yours, you own them, and they are the only tests whose pass/fail rate matters.
It scores audio, not transcripts. A lot of what makes a voice agent feel wrong, long pauses, talk-over, a flat affect on an upset caller, the wrong emphasis on a critical number, does not survive the trip to text. Pipecat's 1.15.0 release this week added TTS expression tags, a Tone model for user emotion, and classifier-based turn completion. None of that lives in a transcript. If your QA pipeline is reading JSON logs, it is grading the wrong artefact.
It runs before launch and before every change. The reason self-improvement is seductive is that it promises to catch regressions after they happen. "After they happen" is already too late. Every model swap, prompt change, knowledge-base update, or vendor platform release should trigger a scored simulation run before it reaches real callers. That is how you ship confidently into a market where the voice-AI category is consolidating fast enough that Salesforce just agreed to acquire Cresta's contact-centre voice agent stack and the platforms under you keep shifting.
Who owns voice-AI quality on your org chart
If you are the PM for a voice agent, you own the quality bar. The vendor owns the agent. The dev team owns the integration. Ops owns the runtime. QA ownership gets fuzzy precisely because the vendor is now so good at pretending to own it.
A clean answer to "who owns quality" looks like this:
| Responsibility | Owner | Why |
|---|---|---|
| The agent's prompts and flows | Vendor or your builder | They build it |
| The test scenarios | You | They encode your business |
| The pass/fail bar on each metric | You | Your tolerance, not the vendor's default |
| Scoring every production call | Independent QA system | Independence from the build loop |
| Filing regressions | Independent QA system | So the vendor can fix them |
| Approving the fix | You | The build loop cannot grade itself |
The last row is the one most teams get wrong. If the same Copilot that wrote the change is the same Copilot that evaluates the change, no one outside the loop has looked.
Where Roark fits
Roark is built for exactly this seat on the org chart. It sits outside the vendor loop and does three things.
It runs simulation tests before launch, dialling your agent over real PSTN and WebRTC with configurable personas, including distinct voices, 45 languages and accents, speech pace, emotional register, and background-noise environments. The agent on the other end can be anything, including the vendor's hosted platform. We have one-click integrations for Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs, and anything else dials in as a number.
It scores every live call on audio-native metrics, 64+ built-in plus unlimited custom ones, looking at pronunciation, emotion, vocal stress, pace and pauses, and interruptions, not just the transcript. When a call fails, Roark files it as an issue automatically, so the vendor loop learns about the regression from an independent source.
And it turns real production failures into replayable regression tests. The call that went wrong last Tuesday becomes a simulation you can run against every vendor model refresh, every self-improvement iteration, every prompt change. That is how you keep the vendor accountable as they keep shipping, and it works whether your agent runs on Cartesia, Regal, Convoso, or anything else.
The relationship with the vendor's own eval tooling is complementary, not competitive. Their tools help them build. Ours help you ship. Both roles are necessary, and the second one cannot be done by the first.
The two questions to ask every vendor this week
If you take one thing from this post into your next vendor call, take these:
- What exactly does your platform measure, and what does a "successful call" look like in your self-improvement loop?
- What happens when my definition of success and your definition disagree?
A vendor that can answer both cleanly has thought about this. A vendor that waves at a dashboard has not. In either case, the answer that keeps you safe is the same: hold the quality bar yourself, measure with an independent system, and let the vendor do what the vendor is good at, which is building.
