Streaming speech recognition

Streaming speech recognition turns audio into text while it is still being spoken. The client holds a connection open, usually a WebSocket, and sends small chunks of audio every few tens of milliseconds; the server returns partial transcripts that update as the sentence unfolds, then final transcripts once a segment is settled. It is the mode behind live captions, dictation and voice agents, and it is judged on latency as much as on accuracy.

Anatomy of a streaming session#

A typical session over a WebSocket (RFC 6455) runs like this. Details vary by provider, but the shape is consistent.

1. Open and configure. The client authenticates and declares the audio format (encoding, sample rate, channels), the language and options such as vocabulary or context.
2. Send audio. Chunks of raw PCM or a compressed codec arrive at roughly the rate they are captured. See audio formats.
3. Receive results. The server returns partials as the hypothesis changes and finals when a segment is committed, often with word timestamps and confidence scores.
4. Close. The client signals end of stream so the server can flush remaining audio into a last final result.

Inside the model#

A batch model sees the whole utterance before emitting anything. A streaming model must emit text from a prefix. Three design choices make that possible.

• Frame-synchronous decoding. Transducers and CTC tie each output to audio already processed, so they naturally emit text as frames arrive. Attention encoder-decoders need modifications to stream.
• Limited-context encoders. Self-attention is masked so each frame attends to past audio and only a small window of future audio, often processed in chunks. The size of that lookahead is a direct trade: more future context improves accuracy and adds the same amount of delay.
• Cached state. The encoder keeps activations from previous chunks so each new chunk is processed incrementally rather than re-running over the whole history. Without caching, cost would grow with the length of the stream.

Some systems also re-decode a completed segment with more context before issuing the final result, so the final can differ from the last partial. That is expected behaviour, not a bug.

Partials, finals and stability#

Partial results let an interface look responsive. They are also unstable: a partial of it's hard to wreck a nice may be revised to it's hard to recognise speech once the next syllables arrive. Words near the end of a partial are the least stable, because the model has the least right context for them.

A final result is emitted when the recogniser decides a segment is complete, usually through endpoint detection: a combination of detected silence, a maximum segment length, and in better systems a model of whether the utterance sounds finished. Rules of thumb for applications:

• Render partials in place, replacing the previous partial, and append finals.
• Trigger actions, database lookups or LLM calls on finals, not partials, unless you can cheaply cancel speculative work.
• For captions, consider showing only the stable prefix of each partial to reduce flicker.

Where the latency comes from#

For captions, the number that matters is how far behind the speaker the text runs. For voice agents, it is the time from the end of the user's speech to the final transcript, because the agent cannot reply sensibly until it has one. A fixed silence timeout of 500 to 1,000 ms, still a common default, can be the largest single item in that budget. See speech-to-text latency and end-of-turn detection.

Writing a streaming client#

The sketch below shows the structure of a client against a generic WebSocket endpoint: one task sends paced audio, another reads results. Message formats differ by provider; consult your provider's documentation for the real schema and authentication.

import asyncio, json, websockets

CHUNK_MS = 50
CHUNK_BYTES = 16000 * 2 * CHUNK_MS // 1000   # 16 kHz, 16-bit mono PCM

async def stream(path, url):
    async with websockets.connect(url) as ws:

        async def send_audio():
            with open(path, "rb") as f:        # raw PCM, no WAV header
                while chunk := f.read(CHUNK_BYTES):
                    await ws.send(chunk)
                    await asyncio.sleep(CHUNK_MS / 1000)   # pace like a live microphone
            await ws.send(json.dumps({"type": "end"}))

        async def read_results():
            async for message in ws:
                result = json.loads(message)
                kind = "final" if result.get("is_final") else "partial"
                print(kind, result.get("text"))

        await asyncio.gather(send_audio(), read_results())

Sending and receiving must be independent. A client that waits for a result before sending the next chunk will stall, because many chunks produce no result at all.

Running streaming at scale#

Each stream holds a connection and a slot of model capacity for its whole duration, so capacity is planned in concurrent streams rather than audio hours. Servers batch chunks from many streams into one GPU call to keep hardware busy while meeting per-chunk deadlines; the scheduling problem is described in concurrency and scaling and GPU inference for speech. Load balancers need sticky, long-lived connections, and deployments must drain sessions gracefully instead of cutting calls mid-sentence.

How Reson8 approaches this#

Reson8 offers real-time streaming alongside batch, served on EU GPUs by its own inference engine, Violin, which uses fused kernels, CUDA graphs and decoding scheduling. For voice agents, the Turns endpoint replaces a fixed silence timer with eager end-of-turn prediction from acoustic and linguistic signal and adds barge-in detection; on the open Pipecat STT benchmark it measured a 326 ms median time from end of speech to final transcript at 1.10% pooled semantic WER. Streaming costs 2 credits per minute. Protocol details are in the documentation.

Common questions#

References#

1. Fette, I., & Melnikov, A. (2011). The WebSocket Protocol. RFC 6455. IETF.
2. ITU-T Recommendation G.711 (1988). Pulse code modulation (PCM) of voice frequencies. ITU.
3. Graves, A. (2012). Sequence Transduction with Recurrent Neural Networks. arXiv:1211.3711.
4. He, Y. et al. (2019). Streaming end-to-end speech recognition for mobile devices. ICASSP.
5. Zhang, Q. et al. (2020). Transformer Transducer: a streamable speech recognition model with Transformer encoders and RNN-T loss. ICASSP.
6. Pipecat: open source framework for voice and multimodal conversational AI. GitHub.