Voice agent latency budget

A voice agent's latency budget is the sum of every delay between the caller going quiet and hearing the first syllable of the reply. In a cascaded agent the main items are the network, the end-of-turn decision, final transcription, the language model's first token and the first chunk of synthesised speech. Well-tuned systems land under about a second; many land between one and two, and the largest single item is often the wait to decide that the user has finished.

Why the budget is tight#

People take turns quickly. Cross-linguistic studies of conversation find that the gap between one speaker stopping and the next starting clusters around 200 milliseconds. An agent cannot match that with a cascade of network calls, but the further it drifts, the more the conversation feels like a walkie-talkie exchange. Past roughly a second of silence, callers start to repeat themselves, ask "hello?", or talk over the reply when it finally arrives, which then triggers barge-in handling and makes things worse.

Where the milliseconds go#

The diagram shows one turn of a cascaded agent after the user stops speaking. Widths are roughly proportional to mid-range values; ranges are in milliseconds.

End of turn#

The simplest agents wait for a fixed period of silence, commonly half a second to a second, before treating the turn as complete. That wait is pure latency on every turn. It also has to be long, because people pause mid-sentence to recall a date or read a number. Predictive turn detectors decide earlier on turns that sound finished and wait longer on ones that do not; see end-of-turn detection.

Final transcription#

A streaming recogniser has most of the transcript ready before the user stops. The remaining cost is processing the last audio chunk and promoting partial results to a final. Some agents start the LLM on a stable partial and discard the result if the final differs, trading compute for time. The measurement choices are covered in speech-to-text latency.

Language model#

Time to first token grows with prompt length and model size, and hosted APIs vary under load. Long system prompts, retrieved documents and full conversation histories all add prefill time. Prompt caching, smaller models for routine turns and keeping history trimmed all help. A tool call adds a full round trip to an external system plus a second LLM pass.

Speech synthesis#

Streaming TTS returns audio before the whole sentence is synthesised. The orchestrator should forward the first clause as soon as it is complete rather than wait for the full reply.

Measuring it properly#

Instrument every turn with timestamps at each boundary: last speech frame detected, turn closed, final transcript received, first LLM token, first TTS byte, first audio frame sent. Record them against a single clock on the server, and separately measure client-side playout if you control the client.

import statistics

def voice_to_voice_ms(turns):
    return [t["first_audio_out"] - t["user_speech_end"] for t in turns]

def summarise(values):
    values = sorted(values)
    pick = lambda q: values[int(q * (len(values) - 1))]
    return {"p50": pick(0.50), "p90": pick(0.90), "p95": pick(0.95),
            "mean": round(statistics.mean(values))}

Measure on real traffic or realistic recordings, not a quiet test room. Noisy audio makes VAD less certain, long answers stress TTS, and busy hours slow hosted LLMs. The end of the user's speech should come from a reference such as a forced alignment or a careful VAD pass, not from the agent's own turn detector, or the detector's delay disappears from the measurement.

Reducing latency#

• Replace the fixed silence timer. A predictive turn detector closes finished turns quickly and waits on unfinished ones. This is frequently the largest single saving.
• Co-locate services. Run STT, LLM, TTS and the orchestrator in the same region. Each extra cross-region round trip costs tens of milliseconds, several times per turn.
• Keep connections warm. Reuse WebSockets and HTTP/2 connections. TLS handshakes on the critical path are avoidable.
• Stream at every boundary. Forward partials, tokens and audio chunks as they appear.
• Trim the prompt. Summarise old turns, load tool schemas only when relevant, and use prompt caching where the provider supports it.
• Cover unavoidable waits. For a slow lookup, say "Let me check that for you" immediately rather than leaving silence.
• Speculate carefully. Starting the LLM before the turn is confirmed saves time but wastes compute and needs clean cancellation when the user keeps talking.

Beware of optimisations that cut latency by cutting accuracy. An aggressive turn detector that interrupts callers mid-sentence, or a smaller STT model that mishears account numbers, makes the agent faster at being wrong.

How Reson8 approaches this#

Reson8's Turns endpoint targets the first two items in the budget together. It predicts end of turn eagerly from acoustic and linguistic signal rather than waiting on a fixed silence timer, and returns the final transcript as part of the same decision. On the open Pipecat STT benchmark (1,000 samples, smart-turn-data-v3.1) the median time from end of speech to final transcript was 326 ms, with 1.10% pooled semantic WER. Inference runs on Reson8's own engine, Violin, on EU GPUs.

Common questions#

References#

1. ITU-T Recommendation G.114 (2003). One-way transmission time. International Telecommunication Union.
2. Stivers, T. et al. (2009). Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106(26).
3. Levinson, S. C., & Torreira, F. (2015). Timing in turn-taking and its implications for processing models of language. Frontiers in Psychology, 6.
4. Pipecat: open-source framework for voice and multimodal conversational AI. GitHub.
5. Reson8. Turns: end-of-turn prediction for voice agents. reson8.dev.