Endpoint detection, or endpointing, decides when an utterance is over. It turns a stream of speech and pauses into bounded segments, and it is the trigger that makes a streaming recogniser issue a final result. Every endpointer trades two errors against each other: waiting too long adds latency to every exchange, and firing too early cuts the speaker off.
Key takeaways
- The classic endpointer is a VAD plus a silence timer. It is simple and predictable, and it charges the full timeout as latency on every utterance.
- People pause mid-sentence for longer than they pause between sentences surprisingly often. Reading out a phone number is the standard counter-example to a short timeout.
- Model-based endpointers use what was said, not just how long it has been quiet, and can fire earlier on complete utterances while waiting on incomplete ones.
- Endpointing is scored on two numbers: latency after true end of speech, and the rate of early cut-offs. Report both.
- In voice agents, endpointing becomes end-of-turn detection: deciding whether the caller has finished and the agent may reply.
The problem endpointing solves#
Streaming speech has no punctuation marks. The recogniser hears a continuous signal with gaps of varying length and must decide which gaps are boundaries. The decision has consequences downstream:
- A dictation app finalises and formats text at each endpoint.
- A voice search or command system closes the microphone and executes the query.
- A voice agent sends the transcript to the language model and starts preparing a reply.
- A captioning system flushes a caption block to the screen.
In each case an endpoint fired too late makes the system feel slow, and one fired too early acts on half a sentence. In a voice agent that means replying to my account number is four one seven when the caller was going to say four one seven two nine eight.
Silence-based endpointing#
The baseline method runs a VAD and starts a timer whenever speech stops. If speech resumes, the timer resets. If the timer expires, the endpoint fires.
The timeout is the single most important parameter. Short timeouts, a few hundred milliseconds, make short commands feel snappy and cut off anyone who hesitates. Long timeouts, a second or more, are safe for dictation and feel sluggish in conversation. Many products end up with a compromise that is wrong for some users all of the time.
Silence endpointing also inherits every VAD weakness. Background noise can hold a segment open indefinitely, so production systems add a maximum utterance length. A quiet speaker's trailing words may be labelled silence, starting the timer early.
Model-based endpointing#
A silence timer ignores the most useful evidence: the words. "I'd like to book a table for" is obviously unfinished; "that's all, thanks" is obviously finished. Model-based endpointers use that evidence.
Classifiers on acoustic and decoder features#
Early work trained a separate classifier to predict end of query. Shannon et al. at Google fed acoustic features into a model that predicted whether the user had finished speaking, and reported lower latency than a VAD-and-timer baseline at similar cut-off rates. Maas et al. at Amazon combined acoustic embeddings with features from the decoder, such as whether the current hypothesis ended in a likely sentence end.
Endpointing inside the recogniser#
With end-to-end models the endpoint can be a token. Chang et al. trained an RNN-T model to emit an end-of-query symbol as part of decoding, so the recogniser itself predicts when the utterance is over, jointly with the words. This removes a separate component and lets the endpoint depend directly on the recognised text.
Prosody and semantics#
Pitch, lengthening of the final syllable and falling intonation all signal completion, and they differ between languages. Semantic completeness matters too: an address is incomplete without a postcode, a date without a month. Turn-detection models for voice agents, such as Pipecat's open-source smart-turn, combine these cues. The distinction between these layers is covered in VAD vs endpointing vs turn detection.
Measuring an endpointer#
Two numbers describe an endpointer, and quoting one without the other is meaningless, because any system can drive either to zero at the cost of the other.
| Metric | Definition | Better when |
|---|---|---|
| Endpoint latency | Time from the true end of speech to the endpoint firing, usually reported as median and 90th percentile | Lower |
| Early cut-off rate | Share of utterances where the endpoint fired before the speaker finished | Lower |
| Word error rate at endpoint | WER of the finalised transcript, which rises when cut-offs drop words | Lower |
| Time to final | End of speech to final transcript delivered, including decoding | Lower |
The true end of speech comes from human labels or forced alignment. Build the test set from real users on the real channel, including numbers, spelled names and hesitant speakers: that is where endpointers fail. See speech-to-text latency for how endpoint delay fits into the wider latency picture.
How Reson8 approaches this#
Reson8's Turns endpoint replaces the fixed silence timer with eager end-of-turn prediction from acoustic and linguistic signal, and includes native barge-in detection. On the open Pipecat STT benchmark (1,000 samples, smart-turn-data-v3.1) it reported a 326 ms median time from end of speech to final transcript, with 1.10% pooled semantic WER, so the earlier endpoint is measured together with the accuracy of what was finalised.
Common questions#
What silence timeout should I use?
There is no universal value. Command interfaces often use a few hundred milliseconds; dictation and form-filling need longer. Measure early cut-off rate and latency on your own users at several settings and choose the point your product can live with.
Why does my recogniser cut people off when they read numbers?
People pause between digit groups, and those pauses often exceed short timeouts. Lengthen the timeout when a number is expected, or use an endpointer that understands the utterance is incomplete.
Is endpointing needed for batch transcription?
Not in the same way. Batch systems see the whole file, so they segment for readability and model context rather than for latency. Endpointing is a streaming problem.
How is endpointing different from end-of-turn detection?
Endpointing decides that an utterance has ended. End-of-turn detection decides that the speaker has yielded the floor in a conversation. A speaker can finish an utterance and keep their turn. See end-of-turn detection.
References#
- Shannon, M., Simko, G., Chang, S.-Y., & Parada, C. (2017). Improved End-of-Query Detection for Streaming Speech Recognition. Interspeech.
- Maas, R. et al. (2018). Combining Acoustic Embeddings and Decoding Features for End-of-Utterance Detection in Real-Time Far-Field Speech Recognition Systems. ICASSP.
- Chang, S.-Y., Prabhavalkar, R., He, Y., Sainath, T. N., & Simko, G. (2019). Joint Endpointing and Decoding with End-to-End Models. ICASSP.
- Li, B. et al. (2020). Towards Fast and Accurate Streaming End-to-End ASR. ICASSP.
- Pipecat. smart-turn: open-source turn detection model. GitHub.