Most production voice agents are a cascade of three models: speech-to-text (STT) turns the caller's audio into words, a large language model (LLM) decides what to say, and text-to-speech (TTS) turns the reply back into audio. Around them sit a transport layer, a turn detector and an orchestrator that streams data between the parts. The design is modular and debuggable, and its main cost is latency, which every stage adds to.
Key takeaways
- The core loop is STT → LLM → TTS, but a working agent also needs transport, voice activity detection, turn detection, interruption handling and tool calls.
- Every component must stream. An agent that waits for each stage to finish before starting the next feels slow even when each model is fast.
- Text is the interface between stages, which makes the system inspectable: you can log, test and replace each part independently.
- STT errors propagate. A misheard account number becomes a wrong tool call, and the LLM cannot recover what the transcript never contained.
- The main alternative is a single speech-to-speech model. It removes hand-offs but gives up much of the control.
The components#
A voice agent is a real-time loop. Audio arrives continuously, the system decides when the user has finished, generates a reply and plays it back, while listening for the user to cut in. The diagram shows the usual cascaded layout.
Transport#
Audio reaches the agent over WebRTC from a browser or app, over a WebSocket from a server, or over SIP and RTP from the telephone network. Transport determines the codec, the sample rate and much of the network delay. Phone audio arrives at 8 kHz in G.711, which changes what the STT model has to work with (see voice agents on the phone).
Voice activity detection and turn detection#
A voice activity detector labels frames as speech or not speech. A turn detector sits on top and decides whether a pause means the user has finished or is only thinking. The two are often confused; VAD vs endpointing vs turn detection separates them. The turn decision is the single largest source of perceived delay in many agents, covered in end-of-turn detection.
Speech-to-text#
A streaming STT model emits partial results while the user speaks and a final transcript once the turn closes. What matters for an agent is not only word error rate but the time from end of speech to a usable final transcript, and accuracy on the tokens the LLM will act on: names, dates, amounts and identifiers.
Language model#
The LLM receives the transcript plus conversation history and a system prompt, and streams back a reply. It may call tools to look up an order or book an appointment. Its time to first token is usually the largest single model delay in the loop.
Text-to-speech#
TTS converts the reply into audio. Streaming TTS starts synthesising from the first sentence or clause, so the user hears the start of the answer while the LLM is still writing the end.
Orchestrator#
The orchestrator is the glue. It holds conversation state, routes frames between components, handles barge-in by stopping playback and trimming the context to what the user actually heard, and manages timeouts and errors. Open-source frameworks such as Pipecat and LiveKit Agents exist largely to provide this layer.
Why everything has to stream#
Consider a naive agent that records the whole utterance, sends it to STT, waits for the full transcript, sends it to the LLM, waits for the full reply, then synthesises all of it. Each hand-off waits for the slowest part of the previous stage. A two-sentence answer can take several seconds to begin playing.
A streaming agent overlaps the stages:
- STT transcribes while the user is still talking, so only the last fraction of a second remains when they stop.
- The LLM starts generating as soon as the final transcript lands, and its tokens are forwarded immediately.
- TTS begins on the first complete phrase and plays audio while later phrases are still being generated.
What the user experiences is the time to first audio, not the time to complete the reply. The latency budget article breaks this down stage by stage.
What flows between stages#
The interfaces are simple, which is the architecture's main strength.
| Boundary | Carries | Typical format |
|---|---|---|
| Transport → STT | Caller audio | 16-bit PCM or μ-law frames, 10–20 ms each |
| STT → orchestrator | Partial and final transcripts, turn events | JSON messages over a WebSocket |
| Orchestrator → LLM | System prompt, history, latest user turn, tool schemas | Chat messages |
| LLM → TTS | Reply text, sometimes with markup for pronunciation | Streamed text chunks |
| TTS → transport | Synthesised speech | PCM, Opus or μ-law, depending on the channel |
Because every boundary is text or plain audio, you can swap an STT or TTS vendor without touching the rest of the system, replay logged transcripts through a new prompt, and read exactly what the model saw when a call went wrong.
Where cascades fail#
Recognition errors compound#
The LLM only sees text. If the caller said my order is INV seven six two five and the transcript reads my order is in B seven six two five, the lookup fails and the agent apologises for an order that exists. Errors on content words matter far more here than aggregate WER suggests; entity accuracy and semantic WER are better guides.
Paralinguistics are lost#
A transcript drops tone, hesitation, emphasis and laughter. The LLM cannot tell a sarcastic "great" from a sincere one. Some systems add emotion or sentiment tags from a separate audio classifier; speech-to-speech models keep this information natively.
Turn-taking is bolted on#
A text LLM has no sense of timing. Deciding when to speak and when to stop is delegated to separate detectors, and the seams show when the agent interrupts a user who paused mid-sentence or talks over a user who has already started to reply.
Latency adds up#
Each stage contributes delay, and the network hops between hosted services add more. Co-locating components in one region, and ideally one data centre, removes a surprising share of the total.
Common variations#
- Speech-to-speech. One model hears audio and produces audio, as in OpenAI's Realtime API or Kyutai's Moshi. See speech-to-speech models vs pipelines.
- Half-cascade. An audio-native LLM consumes speech directly but outputs text to a separate TTS, keeping control over the voice.
- Multilingual routing. A language identifier chooses the STT language, prompt and TTS voice per call or per turn. See multilingual voice agents.
- Hybrid scripted flows. Regulated steps such as identity checks run as fixed dialogue states, and the LLM handles open conversation around them.
How Reson8 approaches this#
Reson8 provides the STT stage of a cascade. Its Turns endpoint combines streaming transcription with eager end-of-turn prediction from acoustic and linguistic signal, instead of a fixed silence timer, and native barge-in detection. On the open Pipecat STT benchmark (1,000 samples, smart-turn-data-v3.1) it recorded 1.10% pooled semantic WER and a 326 ms median time from end of speech to final transcript.
For the identifier problem above, Patterns lets you declare an expected format such as INV[0-9]{3}, and domain adaptation accepts domain text through the API with no retraining. All processing runs on EU GPUs with zero audio retention.
Common questions#
Is a cascaded pipeline outdated now that speech-to-speech models exist?
No. Cascades remain the common choice for production agents that need tool calls, auditable transcripts, a choice of voice and control over each component. Speech-to-speech models are strongest where natural prosody and very low latency matter more than control.
Which component should I optimise first?
Measure first. In many agents the turn detector's silence timeout and the LLM's time to first token dominate. Reducing a fixed 800 ms end-of-speech wait often gains more than switching TTS vendor.
Do I need a separate VAD if my STT does endpointing?
Often not for transcription, but many frameworks still run a lightweight local VAD such as Silero to detect barge-in quickly and to avoid sending silence over the network.
Where should the components run?
As close together as possible. Each cross-region round trip adds tens of milliseconds, and a turn can involve several of them.
References#
- Stivers, T. et al. (2009). Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106(26).
- Skantze, G. (2021). Turn-taking in conversational systems and human-robot interaction: a review. Computer Speech & Language, 67.
- Défossez, A. et al. (2024). Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037.
- Pipecat: open-source framework for voice and multimodal conversational AI. GitHub.
- LiveKit Agents. GitHub.
- Alvestrand, H. (2021). Overview: Real-Time Protocols for Browser-Based Applications. RFC 8825, IETF.