A voice agent can be built as a pipeline of separate models (speech-to-text, a language model, text-to-speech) or as one speech-to-speech model that hears audio and answers in audio. Speech-to-speech models keep tone and timing, remove hand-offs and can respond very quickly. Pipelines give you a transcript at every step, a free choice of components and tighter control over what the agent says. Most production agents in 2026 are still pipelines or half-cascades, and the choice turns on control more than on latency.
Key takeaways
- Pipelines pass text between stages, so every turn is inspectable, testable and replaceable one component at a time.
- Speech-to-speech models preserve paralinguistic information such as emotion, emphasis and hesitation, which a transcript discards.
- Latency favours integrated models, but a well-tuned pipeline with predictive turn detection closes much of the gap.
- Control favours pipelines: pronunciation of names, a specific brand voice, deterministic scripts and audit trails are easier with explicit text.
- Half-cascades, which take audio in and emit text for a separate TTS, are a common middle ground.
Two designs#
Cascaded pipeline#
The cascaded architecture chains a streaming recogniser, a text LLM and a TTS engine, coordinated by an orchestrator. It descends directly from decades of spoken dialogue systems, with the dialogue manager replaced by an LLM.
Speech-to-speech#
A speech-to-speech model represents audio as tokens, typically from a neural audio codec, and is trained to continue a conversation directly in that space, often alongside text tokens. OpenAI's GPT-4o, exposed through the Realtime API, and Kyutai's Moshi are prominent examples. Moshi is full-duplex: it models the user's and its own audio as two parallel streams, so it can listen while speaking and produce backchannels. Earlier research such as dGSLM (Nguyen et al., 2023) showed that dialogue could be modelled from raw audio of two speakers, including overlaps and laughter.
Half-cascade#
An audio-native model consumes speech directly, keeping tone and timing information, but outputs text that a separate TTS voices. This keeps a text record of what the agent said and lets you choose the voice, while removing the separate STT stage.
Side by side#
| Dimension | Pipeline | Speech-to-speech |
|---|---|---|
| Latency | Sum of stages; well-tuned systems land under about a second | No inter-service hand-offs; can be lower |
| Paralinguistics | Lost at the transcript unless added separately | Tone, emphasis and hesitation preserved |
| Observability | Transcript and reply text at every turn | Needs a separate transcript; output text may not match audio exactly |
| Component choice | Any STT, LLM and TTS; swap independently | Tied to one provider's model and voices |
| Domain vocabulary | STT can be adapted to domain terms and identifiers | Limited control over how names and codes are heard |
| Pronunciation control | TTS lexicons and markup | Prompting only |
| Tool calling | Mature, uses the text LLM's function calling | Supported by major APIs; behaviour depends on the model |
| Languages | Pick the best component per language | Whatever the model covers, at whatever quality |
| Cost model | Per component, often per minute and per token | Usually per audio token in and out |
The control argument#
For a customer service or healthcare agent, the ability to see and constrain exactly what happened on each turn often decides the architecture.
- Auditability. Regulated industries need a record of what the caller said and what the agent replied. A pipeline produces both as a matter of course.
- Guardrails. Text replies can be checked before they are spoken, for example to block a price that is not in the catalogue. With audio output, checking happens after the fact or on a parallel transcript.
- Recognition of what matters. An order code or medication name has to be heard correctly for a tool call to succeed. A dedicated STT stage can be adapted to a domain and measured on entity accuracy. See structured identifiers.
- Debugging. When a call goes wrong, a pipeline lets you ask which stage failed. With a single model, the answer is "the model".
The naturalness argument#
Speech-to-speech models win where the conversation itself is the product. They can respond to a sigh, mirror an excited tone, laugh at a joke and handle overlap in ways a cascade cannot, because the cascade never saw those signals. Full-duplex models can also handle turn-taking natively instead of relying on separate detectors. For companionship, language practice, or open-ended assistants, this can outweigh the loss of control.
Is speech-to-speech always faster?#
Not automatically. A speech-to-speech model removes network hops between services and does not wait for a final transcript. But it still has to decide when to respond, still sits behind the same network and telephony path, and a large model's first audio is not instant. On the other side, a pipeline with predictive end-of-turn detection, co-located services and streaming at every boundary gets much closer than a naive cascade. The latency budget is the right tool for comparing them on your own traffic.
A rule of thumb for choosing#
Start from what the agent must get right. If a wrong order number, dosage or account balance is a serious failure, the transcript and the tool call are the product, and a pipeline gives you the most places to measure and correct them. If the agent mainly has to sound present and responsive, and most turns are open conversation, a speech-to-speech model will usually feel better with less engineering. Many teams end up with both: a speech-to-speech front end for small talk and triage, handing over to a controlled pipeline, or a scripted flow, for the steps that touch money, health or identity.
How Reson8 approaches this#
Reson8 builds for the pipeline and half-cascade side: it provides the speech recognition stage. The Turns endpoint addresses the parts of the cascade that most affect latency and naturalness, with eager end-of-turn prediction from acoustic and linguistic signal and native barge-in detection. Text-only domain adaptation and Patterns address recognition of domain terms and identifiers, which is the main place a cascade can be steered.
Common questions#
Will speech-to-speech models replace pipelines?
They are improving quickly and will take a larger share of use cases. Pipelines are likely to remain common wherever auditability, component choice and control over recognition and voice matter, much as modular systems persist in other areas of engineering.
Can I get a transcript from a speech-to-speech model?
Most APIs provide one, often from a separate transcription pass. It may not match exactly what the model understood, which matters when you use it for audit or analytics.
Which is cheaper?
It depends on call length, model choice and how much conversation history is re-sent each turn. Price both on a representative sample of calls rather than per-minute headline rates.
References#
- Défossez, A. et al. (2024). Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037.
- OpenAI (2024). GPT-4o System Card. arXiv:2410.21276.
- OpenAI. Realtime API guide. OpenAI documentation.
- Nguyen, T. A. et al. (2023). Generative Spoken Dialogue Language Modeling. Transactions of the Association for Computational Linguistics, 11.
- Zeghidour, N. et al. (2021). SoundStream: an end-to-end neural audio codec. arXiv:2107.03312.
- Skantze, G. (2021). Turn-taking in conversational systems and human-robot interaction: a review. Computer Speech & Language, 67.