A streaming speech recogniser sends two kinds of output. Partial results (also called interim results or hypotheses) are its best guess at the audio heard so far, and they can change with every new chunk of audio. Final results are segments the recogniser has committed to and will never revise. Partials make an interface feel live; finals are what you store, search and act on.
Key takeaways
- Partials rewrite themselves because later audio disambiguates earlier audio. "I scream" and "ice cream" are the same sound until the next word arrives.
- A final is emitted when the recogniser decides a segment is complete, usually at an endpoint or after a maximum segment length.
- Display partials, act on finals. Triggering business logic from a partial means acting on text that may disappear.
- Two latencies matter: how fast the first partial appears, and how fast the final arrives after the speaker stops.
- Stability is a measurable property. A recogniser that flickers is harder to build on than one that is slightly slower but steady.
Why live text rewrites itself#
A streaming recogniser receives audio in small chunks, typically 20 to 100 ms each, and decodes as it goes. At any moment it holds a hypothesis for everything since the start of the current segment. When a new chunk arrives, the decoder may keep the existing words and append new ones, or it may decide an earlier reading was wrong and replace it.
Revisions happen because speech is locally ambiguous. The acoustic evidence for a word is spread over neighbouring sounds, and the language model's view of what is likely depends on what comes next. Consider a caller reading out a delivery note:
The user sees the parcel's for a few hundred milliseconds before it becomes the parcel is. Nothing went wrong: the recogniser simply had more evidence by then.
Different architectures revise in different ways. CTC and RNN-T models emit tokens frame by frame and tend to append rather than rewrite, although beam search can still swap the leading hypothesis. Attention encoder-decoder models are not naturally streaming; systems built on them often re-decode a sliding window of audio, which can rewrite several words at once. Some servers add a second pass that rescores the whole segment before issuing the final, so the final can differ from the last partial.
What makes a result final#
A recogniser finalises a segment when it decides no further audio will change it. The usual triggers are:
- An endpoint. The speaker pauses, or a model predicts the utterance is complete. See endpoint detection.
- A length cap. Long monologues are cut into segments so that finals keep arriving even without pauses.
- A client request. Many APIs let the client send a flush or close message that forces the current hypothesis to be finalised.
Once a segment is final, the next partial starts from an empty hypothesis for the following segment. Well-designed APIs label each message explicitly, for example with an is_final flag, and give it a segment identifier or start time so the client can replace partials in place.
Handling both in an application#
The rule that prevents most bugs is simple: display partials, act on finals. A few patterns follow from it.
Live captions and dictation#
Show partials in a lighter style and firm them up when the final arrives. Avoid reflowing the whole line on each update; replacing only the tail keeps the text readable. For broadcast captions, the choice between showing partials and waiting for finals is a trade between delay and on-screen corrections.
Voice agents#
In a voice agent, the language model should normally receive the final transcript of the turn. Some agents start work speculatively on a partial to save time, then discard the result if the final differs. This only pays off if partials are stable, otherwise the agent wastes compute and can produce replies to things the caller did not say.
Search, analytics and storage#
Index and store finals only. Storing partials inflates word counts, duplicates phrases and corrupts any downstream metric such as talk time or keyword hits.
finals, partial = [], ""
def on_message(msg):
global partial
if msg["is_final"]:
finals.append(msg["text"])
partial = ""
handle_committed_text(msg["text"]) # business logic here
else:
partial = msg["text"]
render(" ".join(finals + [partial]))
Latency and stability#
Partials and finals each have their own latency, and they answer different questions. See speech-to-text latency for measurement details.
| Measure | What it captures | Who feels it |
|---|---|---|
| Time to first token | Start of speech to first partial word | Users watching live captions |
| Partial lag | How far behind the audio the partial trails | Dictation and captioning users |
| Time to final | End of speech to final transcript | Voice agents, which wait on it before replying |
| Stability | How often partial words are later revised | Anyone building logic on partials |
The tension is real. A recogniser can emit partials sooner by committing to words on less evidence, but those words are revised more often. It can finalise sooner by using a shorter endpoint, but then it risks finalising mid-sentence. Research on streaming models has made both trade-offs explicit: Shangguan et al. measured how often on-device partials were revised, and FastEmit (Yu et al.) trained RNN-T models to emit tokens earlier without a large accuracy loss.
Accuracy of partials is rarely reported, and it should not be compared with the accuracy of finals. The number that matters for a transcript is the word error rate of the finals. The number that matters for a responsive interface is how quickly and how steadily the partials converge on them.
How Reson8 approaches this#
Reson8 offers real-time streaming and a separate Turns endpoint for voice agents. For agents, the figure that matters is time to final, and Turns reports it directly: on the open Pipecat STT benchmark (1,000 samples from smart-turn-data-v3.1), median TTFS, the time from end of speech to final transcript, was 326 ms, with 1.10% pooled semantic WER. Turns finalises from a prediction of end of turn based on acoustic and linguistic signal rather than waiting out a fixed silence timer. Integration details are in the documentation.
Common questions#
Are partial results less accurate than finals?
Usually, yes. A partial is decoded with less right-hand context, so it has had less evidence to work with. Some systems also rescore segments before finalising, which can correct errors the partial contained.
Can a final result ever change?
Not in a well-behaved streaming API: final means committed. Some systems do send a later corrected transcript after a full-session pass, but that arrives as a separate, clearly labelled message, not as a revised final.
Should I turn partials off?
If nothing on screen or in the pipeline uses them, yes. They add message volume and parsing work. Voice agents that wait for the final anyway often run without them, though partials are still useful for detecting barge-in early.
Why do I see the same words twice in my transcript?
The client is appending partials instead of replacing them. Keep a single partial slot that each new partial overwrites, and only append text when a final arrives.
References#
- Graves, A. (2012). Sequence Transduction with Recurrent Neural Networks. arXiv:1211.3711.
- Shangguan, Y., Knister, K., He, Y., McGraw, I., & Beaufays, F. (2020). Analyzing the Quality and Stability of a Streaming End-to-End On-Device Speech Recognizer. arXiv:2006.01416.
- Yu, J. et al. (2021). FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization. ICASSP. arXiv:2010.11148.
- Google Cloud. Speech-to-Text documentation: streaming recognition and interim results. cloud.google.com.
- Pipecat. STT benchmark and smart-turn data. GitHub.