Testing voice agents for FNOL claims intake

In May, Sierra raised $950M at a $15.8B valuation and told the market where the next wave of enterprise voice AI is going. Buried in the announcement was a line every insurance CX and claims leader should have read twice: agents built on Sierra are now powering first notice of loss, claims processing, and sales at carriers. FNOL, the workflow that historically decided whether a policyholder stayed with a carrier or churned, is being handed to voice agents in production.

That is happening as the 2026 Atlantic hurricane season enters its peak weeks, and as insured catastrophe losses cluster closer to $150B a year than the $100B that used to define a bad year. If your voice agent is anywhere near an FNOL queue, "we tested a few sample calls" is not an acceptance test. This post is the QA playbook to replace it with.

Why FNOL breaks voice agents differently

Most voice agent QA plans are written for the easy calls: policy questions, billing balances, ID card requests. FNOL sits at the opposite end of the difficulty curve. The caller has just been in an accident, evacuated a house, or watched a tree come through the roof. They are stressed, sometimes displaced, and often calling from a phone that is not the one on file.

Parloa's insurance buyer's guide describes the shape of the problem cleanly: a single FNOL call can decide whether a policyholder stays or leaves, the queue is long, and your team is already short-staffed during a surge. Layerup puts numbers on the surge itself, noting that catastrophe events can increase claims volume by 3 to 10x within days. Peakflo's catastrophe operations write-up goes further, citing III data that puts the first 72 hours of a major storm at 8 to 12x normal FNOL baseline volume.

That combination, distressed caller plus surge volume plus structured intake requirements, is what makes FNOL the hardest voice AI workflow in the industry, and it is why testing an FNOL agent looks nothing like testing an IVR replacement.

What "good" actually looks like on an FNOL call

Before you can write test cases, you have to agree on what a passing FNOL call looks like. In our experience helping carriers write acceptance criteria, four things carry disproportionate weight.

1. Complete, structured intake. Every downstream system, from adjuster queues to catastrophe vendor dispatch, depends on FNOL fields being captured accurately. Missing a peril type or getting the date-of-loss wrong is not a UX issue, it is a claims-leakage issue.
2. Empathy under stress. Perspective AI notes that modern conversational AI has moved beyond FAQ chatbots to systems that conduct structured interviews, but the caller has to feel heard while that interview happens. Rushing a distressed caller is worse than being slow.
3. Clean escalation. Anyreach's insurance guide puts it well: well-designed deployments treat escalation as a feature, handing off with a structured summary so the licensed adjuster starts with context instead of asking the caller to repeat everything. An FNOL agent that refuses to escalate is worse than one that never picks up.
4. Compliant disclosure and consent. Recording consent, AI disclosure, and PII handling are non-negotiable, and they vary by state.

Each of these needs a metric that a test suite can score, and each needs failing scenarios in your library.

The scenario library you need before launch

Sample-based QA cannot cover FNOL. The distribution of edge cases is too wide, and the tail is where the churn happens. What you need instead is an explicit scenario library, versioned like code, that you run before every launch and every model swap.

A minimum viable FNOL library covers five axes:

• Peril type. Auto collision, homeowner property damage, water damage, theft, fire, wildfire, hail, hurricane, flood. Each has its own required-fields set and its own vocabulary the agent has to recognize.
• Caller state. Calm and organized, mildly distressed, panicked, injured or in pain, intoxicated, hostile, elderly with hearing difficulty. Each state produces different pacing, different interruptions, different fillers.
• Line of business and coverage nuance. Personal auto vs commercial, HO3 vs HO6 vs renters, endorsements the agent has to acknowledge without misrepresenting coverage.
• Environment. From a quiet living room to inside a car with a broken window on a highway, to a shelter with dozens of other evacuees in the background. Noise is the silent killer of ASR accuracy, and it correlates exactly with when your FNOL agent is most needed.
• Adversarial and compliance edges. Callers who refuse recording consent, callers who admit fault they should not admit, callers asking coverage questions the agent cannot legally answer, callers trying to file a fraudulent loss.

You do not need thousands of scenarios on day one. You need every peril crossed with a distressed caller and a noisy environment, and you need every regulated edge case, and you need those scenarios to run automatically before every deploy.

Surge testing is a separate discipline

Single-call quality is table stakes. Surge behavior is the thing that decides whether the agent survives a Cat 3 landfall. Peakflo's catastrophe operations analysis observes that FNOL volume surges within hours of the storm passing, in correlated waves as policyholders exit evacuation and discover damage. Your test plan has to prove the agent handles those waves before the first storm, not during it.

Practically, that means:

• Concurrency ramps. Not "does it work with one caller," but "does p95 latency and metric quality hold at 500, 2,000, or more simultaneous calls." Voice agents that look fine at low concurrency often start dropping barge-ins, mishearing dates of loss, or looping when the STT and LLM providers themselves start throttling.
• Provider-degradation drills. What happens when your primary STT vendor's latency doubles, or your TTS returns a 503? Voice agents are pipelines, and every pipeline has a worst hop. Testing has to include that hop failing.
• Long-call endurance. FNOL calls run longer than the industry average. Agents that leak context after four minutes, or lose the ability to be interrupted after a certain turn count, only reveal that under sustained load.

The insurance industry has been through this movie before with IVRs and human contact centers. What is new is that voice AI degrades in weirder, more subtle ways than either, and only pre-launch simulation over real telephony surfaces those modes.

Audio-native scoring, not transcript grading

The dominant way voice agents get evaluated today is: pull the transcript, hand it to an LLM judge, get a rubric score. For an FNOL call, that misses the point. The transcript will not tell you the agent talked over the caller three times, held a 4.2 second silence after a question about injuries, or delivered a coverage line in an emotionally flat monotone to someone in tears.

The metrics that matter on an FNOL call are audio-native:

• Interruption rate and barge-in handling, especially on questions about injuries or property loss where the caller is likely to jump in.
• Turn latency and dead-air incidents, because a 4-second silence reads as "the AI is broken" to a stressed caller.
• Pace and prosody, because a fast, chirpy delivery on a total-loss call is a churn event.
• Emotional register match, because empathy is not a script line, it is a paralinguistic behavior.
• Pronunciation of addresses, insured names, and peril-specific vocabulary, which STT-then-LLM pipelines routinely mangle in ways the transcript hides.

If you cannot score these on every production call, you cannot claim you are monitoring FNOL quality; you are monitoring a projection of it.

Turning real storm calls into regression tests

The best FNOL test cases are the ones that already broke. After a real storm, the disproportionate value of your production call archive is that it contains every edge your synthetic scenarios missed. Regression on those calls, replayed against the current agent, is what stops the same failure from happening again.

The mechanics of a good replay pipeline:

1. Every production call is scored automatically the moment it ends. Failures are filed as issues with the offending audio and transcript attached.
2. High-value failures get promoted into the scenario library. The caller's audio becomes a persona; the situation becomes a scenario. That scenario now runs before every launch.
3. When the model or prompt changes, the old failing calls are replayed against the new agent. Either the fix holds, or the regression is caught before the change ships.

This loop is the reason voice AI teams that ship weekly can do so at all. Without it, every prompt change is a coin flip on production.

Compliance testing is part of the acceptance test

Consumer sentiment on AI in claims has shifted fast, Insurity's 2026 report shows 51% of consumers say they would feel confident filing a severe-weather claim if AI helped validate the loss, up from 38% in 2025. But confidence collapses instantly if the disclosure is wrong. A few things worth pinning into your acceptance criteria:

• AI disclosure at the top of every call, phrased exactly as your compliance team signed off on. Test that it fires on cold starts, on transfers back from IVR, and on re-connects after dropped calls.
• Recording consent, tested per state. Your test personas need locations, and your suite needs to know which states require two-party consent and enforce that the agent complies.
• PII handling on capture and playback. If the agent reads back a social security number, driver's license, or claim number, the acceptance test needs to prove the caller confirmed identity before the read-back.
• Escalation for coverage questions. The agent should collect the loss, not opine on whether it is covered. Adversarial personas asking coverage questions belong in the library.

Where Roark fits

Everything above is the QA program. Roark is how insurance teams run it without building it themselves. Roark's simulations dial your voice agent over real PSTN and WebRTC, using personas that carry distinct voices, languages and accents, emotional register, and background-noise environments, so an "evacuee on a cell phone in a shelter" scenario actually sounds like one. Runs are triggered from HTTP so they gate CI, and the same platform scores every production call on 64+ audio-native metrics that catch what transcript grading misses. Failing production calls are captured and can be replayed against the updated agent as regression tests, closing the loop we described above.

For carriers, the platform is SOC 2 Type II certified, offers a HIPAA BAA, and exposes OpenTelemetry traces for every call, which matters when your compliance team asks how the agent was tested before a claims-leakage incident. Deeper integration details for Vapi, Retell, LiveKit, Pipecat, Bland, and ElevenLabs live in the docs, and the insurance industry page has more on FNOL, policy servicing, and renewals.

The takeaway

Voice AI in insurance stopped being a pilot and started being production. Sierra's Fortune-50 footprint, the wave of insurance-specific platforms, and the fact that agents on Sierra alone are handling billions of interactions including processing insurance claims all point to the same thing. FNOL is now a voice-AI workflow, and the carriers that treat it as one will win the next storm.

The ones that ship without a real testing program will discover, in the middle of a Cat 3 surge, that their agent was good enough for the demo and not good enough for the queue. FNOL is not the workflow to learn that on.