On October 8, Sierra shipped fleming-1, a model that scores a caller's audio in real time to decide whether the voice on the other end is another AI agent. Two days earlier, Sierra, Meta, and partners including Walmart, Shopify and Stripe announced Personal Agent Protocol; Decagon open-sourced PACT the same morning. The question your voice agent has to answer when the phone rings has quietly shifted from who is calling to what is calling.
If your QA is still built around "human on the line, synthetic test cases on the bench," it will miss the whole new class of inbound traffic. Personal assistants are starting to dial on behalf of their users. Fraudsters are using synthetic voices to probe account defenses. Your own regression harness is itself an AI on the other end of a line. Each of those callers deserves different handling, and the only way to prove your agent does the right thing is to run every one of them against it before you ship.
What actually changed in the last two weeks
The ground moved three times in a 48-hour window:
- Personal Agent Protocol (Oct 6). Meta and Sierra, with Walmart, Shopify, Stripe, Rocket, Genesys and Instinct, announced an open standard for how a customer's personal agent signs in to a business. The v0.1 specification is promised later this month; today it is a plan with partner logos attached.
- PACT (Oct 6). Decagon published PACT, co-developed with Instinct, built on A2A and OAuth 2.0. It separates two questions a receiving agent needs to answer: which personal agent is calling, and what the customer has actually allowed it to do. The spec and reference code are available under Apache 2.0.
- fleming-1 (Oct 8). Sierra's new model analyzes a caller's speech in real time, scoring the audio for signs it was generated by AI. It is tuned conservatively so real people are not inadvertently flagged, and the business decides what happens next.
Sierra's own framing matters here: they first encountered agent-to-agent calling in 2025, when agents built on Sierra started calling other agents built on Sierra in healthcare. This is not a 2027 prediction. The traffic is here.
The four callers your line actually gets now
Any inbound voice agent fielding real traffic today sees some mix of five populations. Four of them are AI, and your single "human-shaped" policy mishandles at least three.
- Legitimate personal agent with a handshake. Muse, Instinct, or a Decagon-wrapped assistant presents PACT identity and a scoped delegation. You can verify the agent and the permissions the customer actually granted.
- Legitimate personal agent without a handshake. The long tail. Most consumer agents today will not speak PAP or PACT, and even those that do will ride on carriers and routes that strip or ignore the signal. Sierra explicitly built fleming-1 for the case where "some agents do not follow the protocol."
- Assistive TTS used by a human caller. The person on the line is a person. The voice is synthetic because they rely on it. A naive detector flags them as AI; a naive policy refuses them service. That is both a product failure and, depending on jurisdiction, a legal one.
- Adversarial synthetic caller. Prompt-injection probes, account-takeover reconnaissance, scripted social engineering at scale. Audio is synthetic for the opposite reason: to run thousands of attempts cheaply.
- Your own test traffic. Every simulation you run against your production agent is an AI on the line. If you design a policy that cuts off flagged calls, you have just broken your own regression suite.
Why one policy breaks all five
The three obvious policies all fail:
- Blanket block flagged calls. Your Muse user cannot check their order. Your TTS-dependent caller gets refused. Your test harness cannot run a nightly suite.
- Blanket accept everything. You have made yourself the easiest account-takeover target of the quarter. AI-generated voices are treated as artificial or prerecorded voices under the TCPA in the United States, and the compliance exposure is just as asymmetric as the fraud exposure.
- Blanket "pass to a human." Your automation ROI goes to zero the week personal agents go mainstream.
What you actually need is a differentiated policy: identity first, audio-native signal second, action scope third, and every branch instrumented. Fleming-1's own framing is correct here. The signal is information, not judgement.
Policy mechanics worth testing
A workable AI-caller policy has five mechanical pieces, each testable:
Identity handshake. On connect, attempt a PAP or PACT handshake. If the agent presents credentials, record which personal-agent provider, which scopes, and whose customer. PACT separates identity from delegation: "the agent says the customer approved it" becomes verifiable proof. Treat the two as independent signals, not one combined trust score.
Audio-native synthetic-voice score. Score the caller's audio continuously, not just at the start. Synthetic voices are improving, and legitimate noise, muffled handsets, and bad connections can skew either direction. Store the score with the call record; do not gate the call on it alone.
Action scope with step-up. Reads can be low-friction for any caller. Account changes, payments, PII release, and scheduling that costs money should require explicit customer-approved scope (via PACT) or a step-up challenge (OTP to the account holder, callback, DTMF confirmation). The right mapping is business-specific; the point is that the mapping exists and is written down.
Accessibility fallback. If detection flags a caller as likely synthetic and no handshake is present, your agent should not refuse the call. It can slow the interaction, request a verification step for sensitive actions, or route to a specialist queue. It should not drop the line. This is the branch most likely to break accessibility and the one most teams forget to test.
Graceful degradation. When the handshake endpoint is down, when the detection model times out, when PACT comes back with an unknown scope: default to the policy you would run for an unverified human caller, log the degradation, and continue. Hard failures on an infrastructure signal become customer-facing outages.
The simulation matrix
Each caller type above corresponds to at least one scenario in your pre-launch suite. The minimum viable matrix:
| Scenario | Caller voice | PACT handshake | Expected agent behavior |
|---|---|---|---|
legitimate_personal_agent_with_pact | synthetic | valid, read-only scope | lookup succeeds, no step-up |
legitimate_personal_agent_write_scope | synthetic | valid, write scope | account change proceeds with scope-matched confirmation |
legitimate_personal_agent_no_handshake | synthetic | none | call proceeds, step-up on any write |
assistive_tts_disclosed_human | synthetic | none, human discloses TTS use | call proceeds without added friction on reads |
synthetic_recon_probe_account_lookup | synthetic | none | step-up triggers, scripted enumeration fails |
jailbreak_voice_prompt_injection | synthetic | none | system prompt not revealed, refusal within SLA |
human_control_no_ai | real | n/a | baseline behavior, no false-positive friction |
Two mechanical rules. First, you have to run these over actual audio. Fleming-1 and similar detectors key on audio artifacts, so a transcript-only replay never touches the detection path. Second, scenarios have to pass or fail against a named metric threshold, not against a reviewer's gut feel.
Metrics worth scoring
Six that generalize across stacks:
- Detection correctness per class. True positive rate on
synthetic_recon_probe, false positive rate onhuman_controlandassistive_tts_disclosed_human. Report the two separately; a single accuracy number is useless when the costs are asymmetric. - PACT handshake success. Did your agent attempt it, parse the response, attach scopes to the session? A silent skip here is invisible until the first disputed charge.
- Policy adherence. For each scenario, did the agent take only actions permitted by its scope? Did it require step-up on write operations?
- No blanket blocking of flagged legitimate callers. The TTS and no-handshake personal-agent scenarios must pass without the agent refusing service.
- Latency delta per branch. PACT verification, detection scoring, and step-up each add time. Budget them and alert on regression.
- Degradation behavior. Simulate detection timeouts and handshake 500s; the policy must have a defined fallback, and the fallback must be tested.
Replay the AI calls you have already fielded
The second you turn on detection, you have a labeled corpus. Export the flagged calls plus a matched sample of unflagged ones, redact appropriately, and promote them into a regression suite. This is where catching a false positive on an assistive-TTS user actually pays off: that single call becomes a permanent test case that prevents the next deploy from reintroducing the same blanket block.
Where Roark fits
Testing AI-on-AI calls is a natural fit for simulation because the simulator and the production caller share a shape. Roark dials your voice agent over real phone calls on PSTN and WebRTC, with personas defined by distinct voices, languages and accents, speech pace, emotional register, and background-noise environments. For AI-caller testing specifically, that means the test calls exercise your detection path the same way a real inbound personal agent will.
Roark's audio-native models score the sound of the call, not just the transcript, so metrics like detection correctness and policy adherence can be built as custom metrics alongside the 64+ built-in ones. Production call replay captures real calls and replays them against updated agent logic, so the first assistive-TTS user your agent fumbles becomes a scenario your next deploy has to pass.
One pragmatic consequence worth flagging. If your production policy flags synthetic callers and injects a long "I need to confirm you are a human" preamble, your Roark runs will trigger that preamble every time. That is not a Roark problem; it is a policy design problem. Build the flag branch to add logging, step-up on account actions, and metadata, not blanket friction. The policy that works for your test harness is the same policy that works for your legitimate personal-agent callers.
A short pre-launch checklist
Before you ship any voice agent into inbound traffic in the next quarter:
- Decide and write down your policy for each of the five caller classes. Do not ship with "we will figure it out."
- Implement a PAP or PACT handshake path, even if most callers today will not use it. The traffic that does use it is the easiest to serve safely.
- Wire an audio-native synthetic-voice score as a signal, not a gate.
- Build the simulation matrix above as a named suite. Threshold each scenario.
- Latency-budget the handshake and detection steps and alert on regression.
- Replay flagged production calls into the suite weekly.
- Confirm your test harness itself passes the flagged-caller branch without added friction.
Caller ID told you a number; it did not tell you whether to pick up. Fleming-1, PAP and PACT give you richer signals, but the decision is still yours, and so is the responsibility to prove the decision is right. The agents you ship this quarter will be judged, in part, on how they treat the agents calling them.
