A voice agent is software that holds a spoken conversation in real time: it listens, works out what the person wants, does something about it, and replies in a synthesised voice. Most production voice agents are a pipeline of three models, speech-to-text, a large language model and text-to-speech, wrapped in logic that decides when the person has finished speaking and what to do when they interrupt. Their quality depends less on any single model than on how fast and how accurately the pieces hand off to each other.
Key takeaways
- The standard architecture is a cascade: STT turns speech into text, an LLM decides the reply and any actions, TTS speaks it.
- Turn-taking is the hardest part to get right. Waiting for a fixed silence makes the agent slow; answering too early cuts people off.
- Latency is cumulative. Every stage adds delay, and people notice pauses much longer than those in human conversation.
- Transcription errors propagate. If the STT mishears an order number or a name, the LLM acts on the wrong input with full confidence.
- Speech-to-speech models are an alternative to the cascade, trading control and transparency for simplicity.
Definition#
A voice agent differs from older voice systems in what it can handle. An interactive voice response (IVR) menu offers fixed options ("press one or say billing"). A voice assistant such as a smart speaker handles one command at a time. A voice agent holds an open-ended, multi-turn conversation, keeps track of context, and can call external systems to look up an order, book an appointment or update a record. The shift came with large language models, which made the "understand and decide" step flexible enough to handle unscripted speech.
Voice agents are used for customer service lines, appointment scheduling, order status, outbound reminders, clinical intake, in-car assistants and internal help desks. Most run over the phone, which brings its own constraints; see voice agents on the phone.
How a voice agent works#
A single turn of conversation goes through five steps:
- Audio capture. Audio arrives from a phone call, browser or app, usually as a continuous stream in small frames. Phone audio is often 8 kHz; see telephony transcription.
- Speech-to-text and turn detection. A streaming recogniser produces partial and final transcripts while the person speaks. Alongside it, something must decide when they have finished: a voice activity detector with a silence timer, or a dedicated end-of-turn model.
- Reasoning and actions. The final transcript, conversation history and instructions go to an LLM, which writes the reply and may call tools such as a booking API or CRM lookup. See tool calling in voice agents.
- Speech synthesis. Text-to-speech converts the reply to audio, usually streaming it sentence by sentence so the agent can start talking before the LLM has finished.
- Playback and listening. While the agent speaks, it keeps listening. If the person interrupts, the agent should stop, discard the rest of its reply, and take the new input.
An orchestration layer ties these together, managing streams, cancellation and state. Open-source frameworks such as Pipecat and LiveKit Agents provide this layer; see voice agent frameworks and voice agent architecture.
What makes voice agents hard#
Knowing when to speak#
Human conversation runs on tight timing. Stivers and colleagues, studying ten languages, found that the most common gap between one speaker finishing and the next starting is around 200 milliseconds or less. People start planning their reply before the other person has finished, using grammar and intonation to predict the end of the turn.
A voice agent that waits for, say, 800 milliseconds of silence before responding feels sluggish, and one that answers after 200 milliseconds of silence interrupts anyone who pauses to think: my account number is four seven one… two nine becomes a reply after four seven one. Better systems predict end of turn from what was said and how it was said, not silence alone. See VAD vs endpointing vs turn detection.
Latency adds up#
Each stage contributes delay: the turn decision, the final transcript, the LLM's first tokens, any tool call, the first synthesised audio, and network hops between them. These add up quickly, and a slow tool call can dominate. Streaming at every stage and overlapping them, for example starting the LLM on a confident partial transcript, is how production systems keep the pause short. See voice agent latency budget and speech-to-text latency.
Interruptions#
People interrupt agents to correct them, to skip a long explanation, or simply to say "mm-hm". The agent has to tell a real interruption from a backchannel or background noise, stop its own audio quickly, and know how much of its reply the caller actually heard. Echo from the agent's own voice on a speakerphone can trigger false interruptions. See turn-taking and barge-in.
Errors propagate#
The LLM only sees the transcript. If INV7625X arrives as INB seven six two five X, the lookup fails, and the agent may apologise, ask again, or confidently act on the wrong record. Names, addresses, codes and numbers matter far more than filler words, so aggregate WER is a weak guide to voice agent quality. Semantic WER and entity accuracy track what matters better, and structured identifier handling is often the single biggest fix.
Cascades and speech-to-speech models#
The alternative to the pipeline is a speech-to-speech model that takes audio in and produces audio out directly, without an explicit transcript in between. It can react to tone and timing, and removes some hand-off delay. The cascade remains the common production choice because each part can be chosen, tested and replaced independently, the text in the middle can be logged, audited and redacted, and tool calling and guardrails operate on text the developer can see.
| Cascaded pipeline | Speech-to-speech model | |
|---|---|---|
| Components | STT, LLM, TTS, chosen separately | One model |
| Transcript | Explicit, loggable | Optional or derived |
| Control | Swap or tune any stage | Limited to what the model exposes |
| Paralinguistics | Lost unless passed separately | Tone and emphasis available |
| Languages and vocabulary | Best STT per language; domain adaptation at STT | Whatever the model supports |
See speech-to-speech models vs pipelines for the full comparison.
How Reson8 approaches this#
Reson8's Turns endpoint is a streaming speech-to-text interface built for voice agents. It predicts end of turn eagerly from acoustic and linguistic signal rather than a fixed silence timer, and detects barge-in natively. On the open Pipecat STT benchmark (1,000 samples from smart-turn-data-v3.1) it measured 1.10% pooled semantic WER, 1.39% mean semantic WER and a median time from end of speech to final transcript of 326 ms. For identifiers such as order numbers, Patterns biases the model toward an expected format. It covers nine European languages, not a hundred.
Common questions#
Is a voice agent the same as a chatbot with a voice?
Architecturally it often is a chatbot with STT in front and TTS behind. In practice, real-time turn-taking, interruptions and latency make it a different engineering problem.
How fast does a voice agent need to respond?
As close to human turn gaps as possible. Pauses much beyond a second are clearly noticeable. Measure from the end of the caller's speech to the agent's first audio.
Do I need a speech-to-speech model?
Not usually. Cascades are easier to debug, audit and adapt to your vocabulary. Speech-to-speech models are worth testing where tone and naturalness matter more than control.
What matters most in the speech-to-text part?
Streaming latency, turn detection, and accuracy on the words your agent acts on: names, numbers and identifiers.
References#
- Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4).
- Stivers, T. et al. (2009). Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106(26).
- Skantze, G. (2021). Turn-taking in Conversational Systems and Human-Robot Interaction: A Review. Computer Speech & Language, 67.
- Pipecat: open-source framework for voice and multimodal conversational AI. GitHub.
- LiveKit Agents. GitHub.