Testing a voice agent means checking that whole conversations succeed under realistic audio, timing and behaviour, not only that each model scores well in isolation. A practical test strategy has layers: component benchmarks for speech recognition and synthesis, text-mode conversation tests for logic and tools, audio simulations with synthetic callers for turn-taking and latency, and monitoring of real calls. Because agents are non-deterministic, each scenario should be run many times and judged on how reliably it succeeds.
Key takeaways
- Component metrics are necessary but not sufficient. A low WER does not tell you whether the agent interrupts callers or books the wrong date.
- Test in layers, from fast and cheap (text-only) to slow and realistic (audio over the real phone path).
- Run each scenario repeatedly. An agent that succeeds four times in five fails one caller in five.
- Measure timing as well as content: voice-to-voice latency, cut-offs and false barge-ins are failures callers notice immediately.
- Turn production failures into regression tests. Real calls are the best source of scenarios you did not think of.
Why voice agents are hard to test#
Three properties separate voice agents from ordinary software. They are non-deterministic: the same input can produce different replies, and occasionally different actions. They are real-time: a correct reply delivered two seconds late, or spoken over the user, is still a failure. And their input is audio, with accents, noise, codecs and hesitation that text tests never exercise.
The classic framework for spoken dialogue evaluation, PARADISE (Walker et al., 1997), already separated task success from dialogue costs such as length, repair turns and recognition errors, and related both to user satisfaction. The same split works for LLM agents: did the caller get what they needed, and what did it cost them to get it?
Layers of testing#
| Layer | What it tests | Speed | Typical metrics |
|---|---|---|---|
| Component | STT on your audio, TTS pronunciation, turn detector | Fast | WER, semantic WER, entity accuracy, cut-off rate |
| Text conversation | Prompt, dialogue logic, tool use, guardrails | Fast | Task completion, tool-call correctness, policy violations |
| Audio simulation | Turn-taking, interruptions, latency, behaviour in noise | Slow | Voice-to-voice latency, false barge-ins, completion under noise |
| Real channel | Telephony path, codecs, carrier delay | Slowest | End-to-end latency, audio quality, call setup failures |
| Production | Everything, on real callers | Continuous | Completion, escalation rate, repeat calls, complaints |
Component tests#
Build a test set of recordings from your own channel and domain, with reference transcripts, and score candidate recognisers on it. General leaderboards measure different audio. Pay particular attention to the values your tools consume: names, dates, amounts and codes. See building your own ASR test set and entity and keyword accuracy. For the turn detector, measure cut-off rate and delay on continuous conversational audio, as described in end-of-turn detection.
Text conversation tests#
Drive the agent's LLM layer directly with text, bypassing audio. These run in seconds and suit CI. Write scenarios as a goal, a caller persona and success criteria, and let a second model play the caller. This is where most prompt and tool bugs are cheapest to find.
Audio simulation#
Put the simulated caller behind a TTS voice and connect it to the agent over the same transport real users use. Now the test exercises VAD, turn detection, STT, barge-in and the latency budget.
Simulated callers#
User simulation has a long history in spoken dialogue research, where simulated users were used to train and evaluate dialogue managers (see Schatzmann et al., 2006). LLMs make simulated callers far more varied. A good suite covers:
- Happy paths for each task the agent supports.
- Difficult input: spelled names, long account numbers, corrections ("no, the fifteenth, not the fifty"), answers to a different question.
- Audio conditions: background noise, 8 kHz telephone audio, accents and non-native speakers, quiet or distant microphones.
- Timing behaviour: long mid-sentence pauses, rapid speakers who leave no gap, interruptions, backchannels during long replies.
- Adversarial callers: off-topic requests, attempts to extract the system prompt, requests the agent must refuse, abusive language.
What to measure#
Outcome#
- Task completion, verified against system state where possible: was the booking made, for the right date?
- Tool-call correctness: right tool, right arguments, no unnecessary or duplicated calls. See tool calling in voice agents.
- Policy adherence: required disclosures spoken, identity verified before account actions, no invented facts.
Conversation cost#
- Number of turns and total duration to complete the task.
- Repair turns: how often the agent asked the caller to repeat or corrected itself.
- Escalation to a human, where that is a designed outcome rather than a failure.
Timing#
- Voice-to-voice latency per turn, reported at the median and p95. See latency budget.
- Cut-off rate: the agent started speaking before the caller finished.
- False barge-in rate: the agent stopped for noise, echo or a backchannel.
- Barge-in response time: how long the agent kept talking after a genuine interruption.
Reliability#
Run each scenario several times. τ-bench (Yao et al., 2024) introduced pass^k, the probability that an agent succeeds on all of k attempts at the same task, and showed that it falls well below single-attempt success for current models. For a voice agent handling thousands of calls, consistency is the property that matters.
Judging conversations#
Use the cheapest reliable method for each criterion. Deterministic checks come first: did the tool run with these arguments, was the disclosure phrase spoken, was latency under the threshold. A model-based judge with a written rubric suits subjective criteria such as whether the agent answered the question or stayed on topic. Keep a human review sample, especially for new scenarios and for any judge you have recently changed.
Judge the transcript the agent actually saw as well as a reference transcript of what the caller said. When the two diverge, the failure is in recognition, not reasoning, and the fix belongs in a different place.
From production back to tests#
No suite anticipates every caller. Log every call with per-turn timestamps, transcripts, tool calls and outcome, subject to your retention and consent rules. Review calls flagged by low completion, long duration, repeated turns or escalation. Convert each confirmed failure into a scenario, and run the suite on every prompt, model or provider change. Where audio cannot be retained, keep the scenario description and transcript, and re-synthesise the caller's side.
How Reson8 approaches this#
Reson8 evaluates its Turns endpoint the way this article recommends for the recognition layer: on the open Pipecat STT benchmark, which uses 1,000 turn-level samples from smart-turn-data-v3.1 and scores both meaning and timing. It recorded 1.10% pooled semantic WER, 1.39% mean semantic WER and a 326 ms median time from end of speech to final transcript. Because Reson8 retains no audio, teams that want recordings for regression testing keep them on their own side.
Common questions#
How many scenarios do I need?
Start with every supported task, a handful of difficult inputs for each, and the timing and audio conditions your callers present. Tens of scenarios run several times each catch far more than one run of hundreds.
Can synthetic voices stand in for real callers?
For logic, turn-taking and latency, largely yes. For recognition accuracy, not fully: TTS audio is cleaner and more regular than real speech. Validate STT on real recordings.
Should tests run in CI?
Text-mode tests, yes, on every change. Audio simulations are slower and cost more; run them nightly and before releases.
References#
- Walker, M. A., Litman, D. J., Kamm, C. A., & Abella, A. (1997). PARADISE: A framework for evaluating spoken dialogue agents. Proceedings of ACL.
- Schatzmann, J., Weilhammer, K., Stuttle, M., & Young, S. (2006). A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The Knowledge Engineering Review, 21(2).
- Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks. arXiv:2306.05685.
- Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045.
- jiwer: evaluate speech recognition with WER, MER, WIL and CER. GitHub.