Concurrency in speech recognition is the number of audio streams or files a system handles at once, and scaling is keeping latency and cost stable as that number grows. Batch transcription scales like any job queue: add workers and the backlog clears faster. Live streaming is harder, because every stream is a long-lived connection that must be served in real time for as long as the caller keeps talking. Capacity planning therefore starts from peak simultaneous streams, not from total minutes per month.
Key takeaways
- Peak concurrency, not monthly volume, sizes a streaming deployment. Little's law converts call arrival rate and duration into concurrent streams.
- A streaming session is stateful and pinned to one server. Load balancing happens when a session starts, not per chunk.
- GPUs serve many streams at once by batching their chunks. More streams per GPU lowers cost until per-chunk latency starts to rise.
- Plan for bursts and for failure: headroom above the expected peak, and clients that reconnect cleanly.
- Watch tail latency. At thousands of streams, the 99th percentile is what a noticeable number of users experience.
Batch and streaming scale differently#
| Batch (pre-recorded) | Streaming (live) | |
|---|---|---|
| Unit of work | A file | A session lasting as long as the audio |
| State | None between jobs | Model and decoder state per stream |
| Can work wait? | Yes, within a turnaround target | No, audio keeps arriving |
| Sized by | Hours of audio per hour, deadline | Peak simultaneous streams |
| Main metric | Throughput, cost per hour | Per-chunk latency at peak |
A batch system that receives a burst can queue it and finish later. A streaming system that receives a burst must either serve every new stream in real time or reject it. That asymmetry drives most design decisions. See real-time vs batch transcription.
Sizing with Little's law#
Little's law (Little, 1961) says that in a stable system the average number of items in the system equals their arrival rate times the average time each spends in it:
For a contact centre, λ is calls per minute and W is average call length in minutes. A centre receiving 60 calls a minute with an average length of 5 minutes has, on average, 300 calls in progress, and so 300 concurrent streams if every call is transcribed live. Record both parties on separate channels and it is 600.
Averages are not peaks. Traffic follows daily and weekly cycles, and events such as outages or marketing campaigns cause spikes. Plan from the busiest interval, typically measured in 15- or 30-minute windows, and add headroom on top. Contact centres already forecast agent staffing this way, with Erlang models built on the same arrival and duration inputs, and those forecasts are a good starting point for speech capacity.
How a streaming service scales#
A streaming speech service is typically a stateless gateway in front of a pool of stateful GPU workers.
Connections#
Live audio usually travels over a WebSocket (RFC 6455) or a gRPC bidirectional stream. Each connection is long-lived: a support call may last twenty minutes. Load balancers must allow long idle timeouts and route by connection, not by request. See streaming speech recognition and integrating a speech-to-text API.
Session affinity#
A streaming model holds state per stream: encoder cache, decoder hypotheses, partial results not yet finalised. Moving that between servers mid-stream is expensive, so each session is pinned to a worker when it starts. Choosing that worker well, by current load rather than round-robin, matters because sessions last minutes and imbalances persist.
Batching across streams#
Each stream sends a chunk every few hundred milliseconds. A worker collects ready chunks from all its streams and runs them through the model together, which is how one GPU serves many streams. The limit is not memory but time: the batch must finish before the next chunks arrive, or latency accumulates. See GPU inference for speech.
Streams per GPU#
How many streams one GPU can serve depends on model size, chunk length, precision and the engine. A useful upper bound comes from real-time factor: a worker with aggregate RTFx of R at a given batch size can process at most R seconds of audio per second, and so at most R live streams. In practice the usable number is lower, because:
- per-chunk latency rises with batch size, and streaming has a latency ceiling;
- chunks from different streams do not arrive in perfect alignment;
- headroom is needed for bursts and for streams moved off a failed worker.
Measure this by ramping simultaneous synthetic streams against one worker while recording per-chunk latency percentiles. The usable capacity is the concurrency just before 95th- or 99th-percentile latency crosses your target.
Autoscaling, bursts and failures#
GPU capacity cannot be created instantly. Starting a worker means acquiring a GPU, loading weights and warming up, which takes from tens of seconds to several minutes. Autoscaling therefore needs to act ahead of demand, using predicted load or a buffer of warm spare capacity, rather than reacting once latency has already degraded. Scale-in has to wait for sessions to end, since pinned streams cannot simply be dropped.
Failures are normal at scale. Clients should reconnect with backoff and resume the stream, accepting that a few hundred milliseconds of audio may need to be resent. Dean and Barroso (2013) describe how, in large distributed systems, rare slow events on individual machines become a regular experience for users; for speech, that shows up as occasional stalled transcripts unless the service detects slow workers and routes around them.
How Reson8 approaches this#
Reson8 offers real-time streaming, batch transcription and a Turns endpoint for voice agents, served from EU GPU infrastructure with its own inference engine, Violin, which uses fused kernels, CUDA graphs and decoding scheduling built for speech. For customers who need capacity inside their own perimeter, on-premise and VPC deployment is available on enterprise plans. Plan limits and connection details are in the documentation.
Common questions#
How do I estimate concurrent streams?
Multiply peak arrivals per minute by average duration in minutes (Little's law), multiply by channels per call if parties are transcribed separately, then add headroom for bursts.
Should I stream audio I already have as a file?
No. Use batch transcription. It is cheaper, usually more accurate, and does not tie up a live session slot.
What happens if I exceed my concurrency limit?
It depends on the provider: new sessions are typically rejected with an error or queued. Handle this explicitly in your client, for example by falling back to batch transcription of the recording after the call.
References#
- Little, J. D. C. (1961). A Proof for the Queuing Formula: L = λW. Operations Research, 9(3).
- Dean, J., & Barroso, L. A. (2013). The Tail at Scale. Communications of the ACM, 56(2).
- Fette, I., & Melnikov, A. (2011). The WebSocket Protocol. RFC 6455. RFC Editor.
- Yu, G.-I. et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI.
- Kubernetes. Horizontal Pod Autoscaling. Kubernetes Docs.