On November 2, Convoso's no-code Voice AI Agent went generally available, built directly into its outbound sales platform with "human-like conversations, sub-second responses, and direct handoffs to human teams." A week later, Cartesia launched Managed Agents, a hosted voice agent platform with telephony, tools, and outbound calling billed at a few cents a minute. Meanwhile Regal's Copilot pitches an even stronger promise: describe what you want, and the platform builds, stress-tests, deploys, monitors, and continuously improves the agent, flagging underperformance and suggesting fixes you can approve in one click.
If you are the person on the hook for whether the agent actually works on real calls, these launches ask you to make a quiet but consequential tradeoff: hand QA to the same vendor that builds, hosts, and updates the agent. That is the trap. A self-improving voice agent is not a self-testing one, and the person who owns the outcome still has to run the loop from the outside.
The pitch is "we do QA for you"
Read any of the current no-code voice AI pages back-to-back and the shape is identical. The platform generates the agent from a description, auto-generates a test suite to stress the guardrails, monitors live calls, surfaces underperforming topics and drop-offs, and recommends prompt or flow edits. Regal describes Copilot as a chat interface that "helps you build, test, and optimize Voice AI agents through guided conversations," with an auto-generated test suite before launch and continuous review of real conversation transcripts after. CX Foundation's writeup of the launch calls out the strategic goal explicitly: own the full lifecycle of agent management and remove the need for dedicated prompt engineers and AI ops resources.
For a PM watching headcount, that is attractive. Spin up an agent in a day, let the platform grade and tune it, ship fixes without a vendor ticket. The problem is not that the platform's loop is bad. The problem is that it is closed.
Three things happen inside the closed loop that only you can see from outside:
- The test cases are generated from the same prompt and knowledge base the agent is generated from. A gap in your spec stays a gap in your tests.
- Scoring runs on the transcript the vendor's ASR produced, with success criteria the vendor's judge model defined. Audio-level failures that never make it into the transcript are not failures the loop can file.
- The improvement suggestion and the approval both live inside the builder. The record of what you shipped, what passed before you shipped it, and what regressed after lives there too, under the vendor's version history.
That last point is the one that quietly kills teams. When something breaks in production three weeks after a Copilot-suggested change, you want an independent answer to "did this pass our tests before it went out?" If the tests, the agent, and the history all belong to the same vendor, you do not have that answer.
What the loop cannot see
A voice agent is not a text agent that happens to speak. The failures that matter most on real calls are audio-shaped: barge-in that cuts the caller off, dead air after a tool call, a wake-word miss when the caller says "wait, hold on," a background noise profile the agent has never heard, pronunciation that works in the demo and collapses on a Scottish accent. A vendor-owned QA loop that scores transcripts misses most of these by construction, because by the time the audio has become a transcript, the problem is gone.
The practitioner literature on production voice AI failures is consistent on this point. The piece "Why AI Voice Agents Fail in Production" catalogs the patterns: the agent "may invent facts, mishear information, forget state, use the wrong business identity, or claim that an action succeeded when the tool failed." The common thread is that the surrounding workflow, data, permissions, tools, fallback paths, and monitoring were not designed for real calls, which is exactly the surface area a transcript-level improvement loop cannot reach.
The second thing the loop cannot see is the gap between the vendor's demo configuration and your production configuration. One recent comparison of no-code voice platforms puts it bluntly: "Latency in the demo is not latency in production," because real deployments add 100 to 200 ms from tool calls and CRM lookups, and voice quality in the demo uses the premium TTS tier while default deployments do not. The platform's own test suite runs against the platform's own happy path. It does not run against your tools, your CRM, your SIP trunk, your language mix, your call volumes at 4pm on a Tuesday.
Third, and most quietly dangerous: silent vendor updates. One builder review of no-code voice platforms notes that no-code builders tend to hide the underlying AI model layer, giving you less control over output quality. When the hidden model gets swapped, or the endpointing threshold is retuned, or a new TTS version ships, you find out when a customer complains. The platform's own QA loop certifies that the platform still passes the platform's tests. It does not certify that your agent still passes yours.
The outside-in QA plan
The way through is not to refuse no-code platforms. It is to keep the QA loop outside them. Five things live on your side of the line.
1. Scenarios owned by the team that owns the outcome
A scenario is not "the agent should handle collections calls." A scenario is "a cardholder calls about a disputed charge, becomes agitated after the first clarifying question, and then asks to set up a payment plan on the rest of the balance." It has a persona, a starting intent, a mid-call shift, and a defined outcome. You write them. The compliance lead writes them. The ops lead writes them. The platform does not write them, because the platform does not know which three scenarios will cost you your license if they fail.
2. Pre-launch simulation over real phone calls
Before any builder change goes live, the scenario library runs against the candidate version of the agent, over the same voice channel real callers will use. This is where the demo-vs-production gap closes: the test is not a text loopback against a mocked endpoint, it is a phone call, with real TTS, real ASR, real network, real tool calls. Roark runs this leg: scenarios dial your agent over PSTN or WebRTC, with distinct caller voices, accents, speech pace, emotional registers, and background-noise environments, across 45 languages and accents. The runs are triggered over HTTP so they can gate a builder release the way unit tests gate a merge.
3. Audio-native scoring, not transcript grading
A production voice AI metric suite has to score the sound of the call. Pronunciation, pace, pauses, interruptions, vocal stress, emotion. Those signals never appear in a transcript cleanly, and if your QA layer only sees the transcript it will miss the failures your callers hear first. The Roark metric suite has sixty-plus of these built in, and unlimited custom metrics on top; you can read the full list in the docs. The point is not the number, the point is that the scoring runs on the audio.
4. Production replay as regression tests
When a real call breaks, it should not just become a filed ticket. It should become a test. Capture the caller side of the audio, label what went wrong, and replay it against every future version of the agent. This is the piece the vendor-owned loop structurally cannot give you, because the vendor's test suite is generated from the prompt, not from what your worst Tuesday actually sounded like. Roark captures production calls and replays them against updated agent logic, which turns one caller's bad day into a permanent guardrail.
5. A versioned pass/fail history you own
Every builder release, every suggested fix, every monthly vendor update ships against the same scenario library and produces the same artifact: a dated report of what passed, what failed, and what regressed. When compliance asks why the agent behaved a certain way in September, you have an answer that does not depend on the vendor's audit trail. This is the piece that lets you keep moving fast on a no-code platform without losing the ability to stand behind the agent in a regulated review.
Who owns which decision
The no-code shift does not remove the PM job, it reshapes it. A rough division of labor that works:
| Decision | No-code platform | PM / QA owner |
|---|---|---|
| Agent prompt and flow authoring | Co-pilot assisted | Approves |
| Guardrail generation | Auto-generated | Reviewed and extended |
| Scenario library | Not touched | Owned |
| Pre-launch pass/fail | Vendor's own suite runs | Independent simulation gates release |
| Live call scoring | Vendor dashboards | Independent audio-native scoring |
| Underperformance detection | Vendor flags transcript-level issues | Owner files audio-level and workflow issues |
| Regression library | None | Owner curates from production replay |
| Release history of record | Vendor | Owner |
The pattern is simple. Let the builder build. Keep the test, the score, and the record on your side of the fence.
The honest tradeoff
There is a version of this argument that comes off as a brief against no-code voice platforms. That is not it. Convoso, Regal, Cartesia, Vapi's visual flow editor, Retell's templates: these are the right tools for most teams. A sales ops lead who can ship a working outbound agent in a week is a strictly better outcome than a six-month engineering project that never gets finished. The point is only that "we handle QA for you" cannot be what you accept in exchange, because the moment the vendor's tests, scores, and history become the only evidence that the agent works, you have handed over the one thing you cannot hand over: the ability to tell your own leadership and your own regulators why you believe in the agent on your number.
The self-improving loop is a real advance. It compresses iteration. It makes non-technical teams productive. It does not replace an outside-in QA loop; it makes one cheaper to run, because the vendor is now doing more of the authoring and you are doing more of the judging.
If you want the outside-in loop without building it yourself, that is what Roark is for: simulation testing over real phone calls, audio-native scoring on every live call, and production replay that turns real failures into repeatable regressions. One-click integrations with Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs mean the loop sits next to whichever builder your team picked, not inside it. The builder gets to keep improving. You get to keep the receipts.
