End-of-turn detection is the voice agent's decision that the user has finished speaking and is waiting for a reply. The simplest method waits for a fixed stretch of silence, which is slow on every turn and still wrong when people pause mid-thought. Modern detectors predict the end of a turn from the words, the intonation and the timing together, so they can answer quickly when a sentence is clearly complete and wait when it is not.
Key takeaways
- Silence is a weak signal. People pause inside turns to think, and often leave almost no gap at the end of one.
- There are two errors: responding too early (cutting the user off) and too late (dead air). Tuning one threshold trades them against each other.
- Useful cues are linguistic (is the sentence syntactically complete?), prosodic (falling pitch, lengthened final syllable) and contextual (did the agent just ask for a 16-digit number?).
- End-of-turn detection is not the same as voice activity detection or ASR endpointing, though it builds on both.
- Evaluate on real conversational audio, measuring both cut-off rate and response delay, not accuracy alone.
The problem#
In conversation analysis, Sacks, Schegloff and Jefferson described turn-taking as organised around transition-relevance places: points where a sentence, phrase or intonation unit is complete and another speaker may take over. Humans predict these points before they arrive, which is how gaps between turns stay around a fifth of a second.
A voice agent has to make the same call from a live audio stream. Consider two utterances that end with the same half-second of silence:
- I'd like to move my appointment to Thursday. The turn is over.
- My account number is four four seven, um… The turn is not over; the user is reading from a card.
A silence timer treats them identically. Set it at 300 ms and the agent interrupts the second caller. Set it at 1,000 ms and the first caller waits a full second on every turn.
Approaches#
Fixed silence timeout#
Run a VAD and declare the turn over after N milliseconds of non-speech. Typical settings are 500 to 1,000 ms. It is simple, predictable and language-independent. It adds its full duration to every response and misfires on long thinking pauses.
Adaptive timeout#
Adjust the timeout by dialogue state. After "What is your postcode?" wait longer; after a yes/no question, shorter. This is cheap and effective when the agent knows what it asked, and it degrades gracefully. Raux and Eskenazi's work on the Let's Go bus information system reported lower latency at a comparable rate of cut-ins when thresholds depended on dialogue state.
Text-based turn models#
Feed the running transcript to a classifier or small language model that estimates the probability that the utterance is complete. TurnGPT (Ekstedt and Skantze, 2020) is a well-known research example. These models capture syntax well: "to Thursday" is complete, "to" is not. They miss prosody, and they inherit any transcription errors and latency of the STT stage.
Audio-based turn models#
Classify the raw audio directly, learning intonation, final lengthening and breathing patterns. Voice Activity Projection (Ekstedt and Skantze, 2022) predicts future speech activity for both speakers from audio. Pipecat's open-source Smart Turn model is an audio classifier trained to decide whether a speech segment is complete. Audio models hear a rising, unfinished tone that a transcript cannot show, but can struggle with flat or non-native intonation.
Combined acoustic and linguistic models#
The strongest systems use both: what was said, and how it was said. Integrating the turn decision into the recogniser itself also removes a hand-off, since the model that knows the words also knows when they stopped.
Cues that signal the end of a turn#
| Cue | Signals completion | Signals continuation |
|---|---|---|
| Syntax | Complete clause, answered question | Trailing preposition or conjunction: "and…", "because…" |
| Pitch | Falling contour on a statement | Level or slightly rising "list" intonation |
| Timing | Lengthened final syllable, then release | Filled pause: "um", "uh", "euh" |
| Content | Expected slot filled: full date, 8-digit code | Partial slot: "four four seven" |
| Breathing | Exhalation and relaxation | Audible in-breath |
Gravano and Hirschberg found in task-oriented dialogue that the likelihood of a turn change rises sharply as more of these cues co-occur. No single cue is reliable on its own.
Evaluating a turn detector#
Accuracy on a balanced set of "complete" and "incomplete" clips is a start, but it hides the operational trade-off. Measure on continuous conversational audio:
- Cut-off rate: the share of turns where the agent responded before the user finished.
- Response delay: time from the true end of speech to the turn decision, at the median and p95.
- Transcript quality at the decision point: a fast decision is useless if the transcript delivered with it is incomplete. Semantic WER on the final transcript captures this.
Plot cut-off rate against median delay for several thresholds. The curve, not a single point, describes the detector. For how these errors show up in whole conversations, see testing voice agents.
How it relates to VAD and endpointing#
These three are often used interchangeably, and they should not be. Voice activity detection answers "is anyone speaking in this frame?". Endpointing in ASR answers "is this utterance's audio complete enough to finalise the transcript?". End-of-turn detection answers "does the user now expect me to respond?". A turn can contain several endpointed utterances, and a VAD-silent frame can sit in the middle of a turn. The distinctions are laid out in VAD vs endpointing vs turn detection.
How Reson8 approaches this#
Reson8's Turns endpoint makes the end-of-turn decision inside the streaming recogniser. It predicts end of turn eagerly from acoustic and linguistic signal instead of a fixed silence timer, and emits the final transcript with the decision. On the open Pipecat STT benchmark (1,000 samples, smart-turn-data-v3.1) it recorded 1.10% pooled semantic WER, 1.39% mean semantic WER and a 326 ms median time from end of speech to final transcript. Details are in the documentation.
Common questions#
What silence timeout should I use?
If you must use a fixed one, 500 to 800 ms is a common starting range for conversational turns. Lengthen it when collecting numbers, addresses or spelled names, and measure cut-off rate on your own calls.
Can the LLM decide when the user has finished?
It can, and some systems ask the LLM to output a "wait" token when the transcript looks incomplete. It adds a full LLM round trip to the decision and sees no prosody, so it works best as a second check rather than the primary detector.
Does end-of-turn detection work on phone audio?
Yes, but 8 kHz narrowband audio carries less pitch detail, and line noise confuses VAD. Evaluate on telephony recordings rather than wideband studio clips.
What about users who never pause?
Some speakers run sentences together with no gap. A linguistic model helps here, because it can recognise a complete request even when the audio never goes silent.
References#
- Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4).
- Raux, A., & Eskenazi, M. (2009). A finite-state turn-taking model for spoken dialog systems. Proceedings of NAACL-HLT.
- Gravano, A., & Hirschberg, J. (2011). Turn-taking cues in task-oriented dialogue. Computer Speech & Language, 25(3).
- Ekstedt, E., & Skantze, G. (2020). TurnGPT: a Transformer-based language model for predicting turn-taking in spoken dialog. Findings of EMNLP.
- Ekstedt, E., & Skantze, G. (2022). Voice Activity Projection: self-supervised learning of turn-taking events. Interspeech.
- Skantze, G. (2021). Turn-taking in conversational systems and human-robot interaction: a review. Computer Speech & Language, 67.
- Pipecat Smart Turn. GitHub.