On October 6, Hume shipped a new Expression Measurement API that returns 414 vocal expression tags and 190 voice descriptors across 50+ languages, four days after it told customers its EVI and TTS APIs will shut down on November 13. The signal is clear: Hume is leaving the empathic-agent business and leaning into measurement. The downstream signal is clearer still. Emotion detection is being decoupled from any one voice stack and bolted in front of agents built on everything else: LiveKit, Pipecat, Vapi, Retell, you name it.
Which means a lot of production voice agents are about to start making routing decisions based on how a caller sounds, not just what they said. "She sounded frustrated, so we escalated." "He was calm, so we pushed the upsell." "The pitch spike at second nine triggered the de-escalation prompt." These are the kinds of behaviors that look great in a demo and break in nasty, invisible ways in production, because almost none of them are tested.
What emotion-adaptive routing actually does
In the typical production shape, a parallel branch of the pipeline consumes the caller's audio and emits a stream of affect signals. Pipecat's ParallelPipeline forks the frame stream so a conversation branch and a scoring branch see the same transcripts without the scoring branch adding latency to the response. In LiveKit Agents the same pattern is usually implemented as a second agent process or an event handler like on_user_speech_committed that calls out to a sentiment service and pushes results onto a shared state.
Those signals then feed one of a handful of agent policies:
- Hard escalation. Transfer the call to a human queue, often a specialist one. Yellow.ai and similar platforms describe smart handovers triggered by sentiment or confidence drops, with context preserved.
- Tone and pace shift. The LLM gets a system-prompt mutation ("the caller sounds distressed, respond slowly and acknowledge first") or the TTS switches voice preset.
- Flow bypass. Skip the upsell, the CSAT survey, or the ID re-verification when the caller is already upset.
- Routing to a different skill queue. Match severity to agent skill: high-stakes emotional situations go to de-escalation specialists, technical frustration goes to product experts.
Each of these is a branch in the call graph. Each branch has its own prompt, its own tool access, and its own handoff format. None of them get exercised by the sunny-day scenarios that pass through most regression suites.
The failure modes you can't catch from a transcript
The reason emotion-adaptive routing is hard to test is that the input the policy actually consumes is audio, not text. "I would like to cancel my policy" reads neutral. Spoken flat, with a sigh at the end and a 1.2-second pause before the verb, it is a classic churn signal. Spoken crisply with upward pitch at the end, it is a question. Any test that only feeds text into your agent is testing a different code path than the one that will decide whether to escalate.
The specific failures we see:
- False-positive escalations. The model flags measured frustration as rage and dumps the caller into a de-escalation queue they did not need. Agents start reading acknowledgment prompts to callers who were simply asking a question, which annoys them and manufactures the frustration you were trying to pre-empt.
- False-negative escalations. Cold, polite anger passes the detector because it uses the vocabulary of a happy customer. The policy fires at second 180 of a call that should have escalated at second 20.
- Latency injection. The emotion branch is "parallel" on paper, but the agent policy waits for its result before responding. Suddenly your p95 time-to-first-token goes up 300 ms on exactly the calls where responsiveness matters most.
- Handoff that drops context. The escalation fires, but the summary passed to the human is wrong, truncated, or missing the detected emotion label, so the human picks up blind.
- Tone shift that never happens. Detector returns
anger: 0.82, the system prompt is supposed to mutate, and nothing changes in the TTS output because the mutation path has a bug nobody notices. - Threshold drift. Someone retunes the escalation threshold from 0.7 to 0.6 to catch more cases. Three unrelated flows regress because their guard conditions assumed the old threshold.
A text-only evaluation suite catches zero of these. A production dashboard catches most of them weeks after the regression ships.
Why text-driven evals miss the whole category
The temptation is to test the detector in isolation: feed it a WAV file, assert on the output. That is useful for the detector's own CI, and it is approximately zero coverage of the agent that consumes those scores.
What you actually need to assert is end-to-end behavior over a real call: the caller speaks in a specific emotional register, the pipeline routes audio through the real STT, the real detector, the real policy, and the real agent, and the agent takes the right action, with the right tone, in the right amount of time. That is a telephony test, not a transcript test.
Three things fall out of that requirement:
- The caller audio needs to be synthesized or played back with controlled prosody. Reading a script into Deepgram with your laptop mic is not it, because the voice does not vary.
- The call needs to run over the same transport your production uses, PSTN or WebRTC, so you get the same codecs, the same packet timing, the same VAD behavior.
- Assertions need to cover acoustic properties of the agent's response, not just the words. "Did the agent slow down after detecting distress" is a pace question, not a token question.
A sketch of the test harness
The scenario definition for an emotion-adaptive flow looks less like a prompt and more like a casting brief. Something like:
scenario: billing_dispute_escalates
persona:
voice: female_mid_30s_us
emotional_register: rising_frustration
pace: fast
background: office
turns:
- caller_opens: "I have been charged three times for the same plan."
- caller_follow_up: "I called yesterday and nobody fixed it."
expectations:
escalation.fired: true
escalation.fired_by_turn: <= 3
escalation.target_queue: billing_specialists
escalation.handoff_summary_contains:
- triple_charge
- prior_contact
agent.pace_after_detection: slower_than_baseline
agent.first_token_latency_ms: < 1200You run that scenario by dialing the agent, playing the persona's audio, and recording the full call for scoring. In CI, the suite runs on every merge that touches the policy, the detector plug-in, or any prompt in the escalation path. For a Pipecat deployment, a sentiment processor dropped into the pipeline is cheap; validating that the processor's output actually changes agent behavior under real audio is where simulation earns its keep. The Pipecat example of inserting a sentiment_analyzer processor is useful starting code, and useless as a test.
Trigger the whole thing from CI with an HTTP call to your simulation runner. The run blocks until every scenario has either passed, failed, or timed out; failures gate the deploy.
Metrics that are worth tracking
For both pre-launch suites and production monitoring, the metrics that keep an emotion-adaptive agent honest are a small set, measured over audio:
- Escalation precision and recall against a labeled population. Precision keeps you from flooding your human queues. Recall keeps you from stranding callers who needed a human.
- Time to escalation on calls where escalation is the correct outcome. Mean and p95. If a frustrated caller waits 90 seconds to be heard, the detector did not help.
- First-token latency delta on calls where the emotion branch fired. If the branch is costing you 300 ms on hot calls, you need to know.
- Post-detection tone shift, measured on the agent's own audio. Did pace actually drop, did pitch flatten, did the agent acknowledge before informing. Transcript-level evals can't see any of this.
- Handoff summary completeness. For every escalated call, score the summary passed to the human against the content of the call. This is the metric that catches silent regressions in your transfer tool.
- False-escalation rate on neutral intents. Pick a corpus of intents that should never escalate (balance inquiry, hours of operation) and track the escalation rate on them over time.
Where Roark fits
This is exactly the shape of problem Roark was built for. Simulation runs dial your agent over real PSTN or WebRTC, with personas that encode voice, language, pace, emotional register, and background noise. The same suite that proves the policy before launch can be scheduled to run against production nightly, so threshold drift and prompt regressions get caught by the same assertions that gated the release.
On the live side, every production call is scored on 64+ built-in metrics (plus custom ones you define for your routing-specific assertions), and the audio-native scoring looks at pronunciation, pace, pauses, interruptions, and vocal stress, not just the transcript. When an escalation misfires, Roark files it as an issue and lets you replay the exact call against the updated policy, turning real production failures into regression tests. One-click integrations for Vapi, Retell, LiveKit, Pipecat, Bland and ElevenLabs mean you plug the harness into whatever stack your sentiment branch runs on top of.
The thing we did not solve for anyone, and nobody can solve for you, is choosing the right policy. Whether to escalate at anger: 0.6 or anger: 0.8 is a product decision that depends on your humans' capacity and your tolerance for angry handoffs. What simulation and audio-native scoring do is let you make that call with evidence instead of guesswork, and catch it fast when the ground shifts under you.
The uncomfortable truth about emotion-aware agents
The reason emotion-adaptive routing is spreading faster than it is being tested is that it feels like a soft feature, the kind of thing where "it mostly works" is good enough. It is not. An agent that misroutes an angry churn-risk caller to the upsell flow is doing more damage than an agent with no emotion detection at all. Shipping behavior that depends on acoustic signals without testing it on real acoustic signals is a bet on luck.
The teams that will do this well in the next six months are the ones treating emotion-adaptive routing as a first-class testable surface: scenarios defined with emotional register, assertions written against audio properties, suites that gate CI, and production monitoring that scores the same metrics the simulations scored. The teams that treat it as a UX nicety will spend the next six months learning why their escalation queues are full of calm callers and their angry callers never got a human.
