Speech-to-text latency is the delay between someone speaking and the text of what they said being available to use. It is not one number. The two that matter most are how quickly the first partial words appear, and how long after the speaker stops the final transcript arrives. The second is usually what users feel, and it is dominated less by model speed than by how the system decides that speech has ended.
Key takeaways
- Distinguish time to first partial, per-word lag, and final latency after end of speech. Vendors quote different ones.
- Final latency is often dominated by endpointing: a fixed silence timeout of several hundred milliseconds adds that delay to every turn.
- Model throughput (real-time factor) and latency are different properties. A model can process an hour in seconds and still respond slowly to a live stream.
- Report medians and tail percentiles (p95, p99), measured at the client, on your network.
- Latency and accuracy trade off: less lookahead means faster but less certain output.
What "latency" means#
In streaming recognition the client sends audio in small chunks and the server returns partial and final results. Several delays can be measured on that stream:
- Time to first partial: from speech onset to the first interim text. Governs whether a live caption feels responsive.
- Partial lag: how far behind the audio the partial text runs while someone is speaking, typically measured per word from its spoken end time to its first appearance.
- Final latency: from the end of speech to the final, stable transcript. Sometimes abbreviated TTFS. This is what a voice agent waits on before it can reply.
- Batch turnaround: for pre-recorded audio, the wall-clock time from upload to finished transcript. Here throughput matters more than per-word delay.
Where the milliseconds go#
End-to-end latency is the sum of several stages. Each can be measured separately, and the largest is often not the model.
| Component | What sets it | Typical lever |
|---|---|---|
| Capture and chunking | Client buffer size, often 20–100 ms per chunk | Send smaller chunks |
| Network | Round trip to the data centre, TLS, jitter | Region choice, persistent connections |
| Model lookahead | Right context the encoder waits for | Model design; trades accuracy |
| Compute and queueing | GPU scheduling, batching, load | Inference engine, capacity |
| Endpointing | Silence timeout or turn model | Predictive end-of-turn detection |
| Post-processing | Punctuation, formatting, redaction | Run on finals only, or stream it |
Endpointing usually dominates#
A recogniser cannot issue a final result until it believes the speaker has stopped. The simplest rule is a silence timer: after a fixed period of no speech, close the segment. Every millisecond of that timer is added to the final latency of every utterance. Set it short and the system cuts people off mid-thought when they pause; set it long and every response feels sluggish. Endpoint detection and end-of-turn detection models replace the fixed timer with a prediction from acoustic cues (falling pitch, lengthened final syllables) and linguistic cues (whether the sentence is grammatically complete), so the system can finalise quickly after "cancel my order" and wait after "my order number is".
Lookahead and accuracy#
Streaming encoders see only a limited window of future audio. More lookahead gives the model more evidence and fewer errors, at the cost of delay. Research on streaming transducers, such as Google's on-device work from 2019 and 2020, treats this as an explicit trade-off and reports both word error rate and latency for each configuration. Any vendor comparison that reports only one of the two is incomplete.
Latency is not throughput#
Real-time factor measures how much audio a system processes per second of compute. An RTFx of 1,000 means an hour of audio in under four seconds. That describes batch capacity and cost, not responsiveness. A live stream arrives at real-time speed no matter how fast the model is, so streaming latency is bounded by lookahead, network and endpointing. High throughput helps latency indirectly: a fast model spends less time per chunk and queues less under load, which improves tail percentiles.
Measuring latency properly#
- Measure at the client. Server-side timings exclude network and client buffering, which users feel.
- Stream in real time. Sending a file faster than real time makes latency look better or worse depending on how the server buffers. Pace chunks to the audio clock.
- Use reference timestamps. Mark end-of-speech from a forced alignment or careful annotation of the test audio, not from the recogniser's own timestamps.
- Report distributions. Give p50, p95 and p99. A median of 300 ms with a p99 of 2 s produces a noticeably worse experience than a steady 400 ms.
- Pair with accuracy. A faster final that is wrong more often is not an improvement. Report WER or semantic WER on the same run.
How Reson8 approaches this#
Reson8's Turns endpoint is streaming STT built for voice agents. It predicts end of turn eagerly from acoustic and linguistic signal instead of waiting on a fixed silence timer, and detects barge-in natively. On the open Pipecat STT benchmark (1,000 samples from smart-turn-data-v3.1) it measured a median 326 ms from end of speech to final transcript, with 1.10% pooled semantic WER on the same run.
Resonant-1 is served on Reson8's own inference engine, Violin, which uses fused kernels, CUDA graphs and decoding scheduling to reduce per-chunk compute. Its RTFx of 1187 on the Open ASR Leaderboard is a throughput figure, relevant to batch and to behaviour under load, not a streaming latency figure.
Common questions#
What is a good speech-to-text latency?
For live captions, partials within a few hundred milliseconds of the spoken word feel immediate. For voice agents the figure that matters is final latency after end of speech, and every stage after it (the LLM and text-to-speech) adds more, so a few hundred milliseconds is a practical target for the recognition part.
Why is my final transcript slower than my partials?
Because the final waits for endpointing. Partials are emitted as soon as the model has a guess; the final is emitted only once the system decides the utterance is over. A long silence timeout is the usual cause.
Does a faster model always mean lower latency?
No. Streaming latency is bounded by the audio arriving in real time, the model's lookahead and endpointing. A faster model mainly reduces compute time per chunk and queueing under load.
Should I measure latency with pre-recorded files?
Yes, but stream them at real-time pace. Pushing a whole file at once measures batch throughput, not what a live user experiences.
References#
- ITU-T Recommendation G.114 (2003). One-way transmission time. International Telecommunication Union.
- Graves, A. (2012). Sequence Transduction with Recurrent Neural Networks. arXiv:1211.3711.
- He, Y. et al. (2019). Streaming End-to-end Speech Recognition for Mobile Devices. ICASSP.
- Sainath, T. N. et al. (2020). A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency. ICASSP.
- Fette, I., & Melnikov, A. (2011). The WebSocket Protocol. RFC 6455, IETF.
- Pipecat: open-source framework for voice and multimodal conversational AI. GitHub.