Voice AI is the umbrella term for software that listens to speech, speaks back, or does both. The full stack runs from audio transport, through detecting speech and turns, speech-to-text, a reasoning layer that is now usually a large language model, and text-to-speech, back to audio. Most production systems chain separate models for each stage; a newer class of speech-to-speech models collapses them into one.
Key takeaways
- Voice AI covers both input (speech recognition) and output (text-to-speech), and the orchestration between them.
- The dominant design is a cascade: speech-to-text, language model, text-to-speech. Each part can be chosen, tested and replaced on its own.
- In a conversation, latency is as important as accuracy. People expect a reply within a few hundred milliseconds of finishing a sentence.
- Turn-taking, knowing when a caller has finished and when they interrupt, is a separate problem from transcription and often the hardest part to get right.
- Transcription errors propagate. A misheard account number becomes a wrong database lookup, however good the language model is.
Definition#
"Voice AI" is a product and industry term rather than a research field. It groups every system in which a machine processes or produces the human voice using learned models. That includes one-directional tools such as dictation software and audiobook narration, and two-directional systems such as phone agents and in-car assistants.
The term overlaps with "speech AI" and "conversational AI". The distinctions are covered in voice AI vs speech AI vs conversational AI. In short: speech AI names the underlying technologies, conversational AI names the dialogue behaviour, and voice AI names the product experience of talking to a machine.
The stack, from audio in to audio out#
A two-way voice system has more moving parts than the three models usually drawn. Each layer below is a place where latency accumulates and where things can fail.
Transport#
Audio arrives over a phone line (SIP, often at 8 kHz), a WebRTC session in a browser or app, or a WebSocket from a device. The transport sets the audio quality the rest of the stack has to work with. See voice agents on the phone.
Speech detection and turn-taking#
Voice activity detection separates speech from silence and noise. On top of it, end-of-turn detection decides when the user has finished and it is the system's turn to speak, and barge-in handling decides what to do when the user interrupts. A fixed silence timer is the simplest approach and the most common cause of agents that either cut people off or leave long awkward gaps.
Speech-to-text#
A streaming recogniser turns the user's audio into text as they speak. Its errors are the ceiling on everything downstream: if my postcode is 1017 CE arrives as my post code is ten seventeen see, the agent cannot look up the address.
Reasoning#
A language model, sometimes with retrieval and tool calling, decides what to say or do. Before large language models this layer was a hand-built dialogue manager with intents and slots. It is now usually a general LLM given a system prompt and a set of tools.
Text-to-speech#
Text-to-speech converts the reply into audio. Streaming TTS starts speaking after the first few words are generated rather than waiting for the complete reply, which saves hundreds of milliseconds.
Cascades and speech-to-speech models#
The cascaded design has practical advantages. Each component can be benchmarked separately, swapped for a better one, and inspected: the transcript and the reply text are logged, which matters for debugging and for regulated industries that must keep records. Its cost is latency, since each hand-off adds delay, and the loss of paralinguistic information such as tone, which the text layer discards.
Speech-to-speech models, such as Kyutai's Moshi and the realtime modes offered by several LLM providers, take audio in and generate audio out. They can respond faster and react to how something was said, not only what. They are harder to evaluate, harder to constrain, and give less control over vocabulary and voice. The trade-offs are covered in speech-to-speech models vs pipelines.
| Cascaded pipeline | Speech-to-speech | |
|---|---|---|
| Components | STT, LLM, TTS, chosen separately | One model |
| Inspectable text | Yes, at every step | Only if the model also emits text |
| Domain vocabulary control | At the STT and prompt layers | Limited |
| Latency | Sum of stages, reducible with streaming | Potentially lower |
| Tone and emotion | Mostly lost at the text step | Can be preserved |
Why latency dominates#
Human conversation runs on tight timing. A cross-language study of turn-taking by Stivers and colleagues found that the typical gap between one speaker finishing and the next starting is around 200 milliseconds, with small variation across languages. Voice systems do not reach that, but the closer they get, the more natural they feel. A reply that takes two seconds reads as hesitation or failure.
The delay is a sum: detecting the end of the turn, finalising the transcript, the LLM's time to first token, TTS time to first audio, and network round trips on both sides. The voice agent latency budget breaks this down stage by stage.
Where voice AI is used#
- Customer service. Phone agents that answer, route and resolve calls, and analytics on human-handled calls. See contact centre speech analytics.
- Healthcare. Dictation and ambient scribes that turn a consultation into a clinical note.
- Automotive. In-car assistants that must work over road noise. See automotive voice recognition.
- Media and accessibility. Live captions, dubbing and screen readers.
- Devices. Smart speakers and wearables using wake words to decide when to start listening.
How Reson8 approaches this#
Reson8 works on the listening side of the stack. Its Turns endpoint combines streaming speech-to-text with end-of-turn prediction from acoustic and linguistic signal, rather than a fixed silence timer, and native barge-in detection. On the open Pipecat STT benchmark it measured 1.10% pooled semantic WER and a 326 ms median time from end of speech to final transcript. It does not provide the language model or text-to-speech layers; those come from the rest of the pipeline, often assembled with a voice agent framework.
Common questions#
Is voice AI the same as a voice assistant?
A voice assistant such as a smart-speaker assistant is one kind of voice AI product. The term also covers one-way tools like transcription and narration, and business systems like phone agents.
Do I need all the layers of the stack?
Only for two-way conversation. A transcription product needs capture and speech-to-text; a narration product needs text-to-speech. Turn detection and barge-in matter only when the system talks back.
Which part of the stack causes most failures?
In deployed phone agents, the common failures are turn-taking (cutting people off or waiting too long) and misrecognised specifics such as names, numbers and codes. Both sit on the listening side, before the language model sees anything.
Will speech-to-speech models replace pipelines?
They are improving quickly and suit open conversation. For tasks that need auditable text, strict vocabulary or specific voices, cascades remain the practical choice, and many teams run hybrids.
References#
- Stivers, T. et al. (2009). Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106(26).
- Weizenbaum, J. (1966). ELIZA: a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1).
- Défossez, A. et al. (2024). Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037.
- Shen, J. et al. (2018). Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Tacotron 2). ICASSP.
- Pipecat: open source framework for voice and multimodal conversational AI. GitHub.