GPU inference for speech models is the engineering of running a trained recogniser as fast and as cheaply as possible on graphics processors. The model's arithmetic is rarely the bottleneck. Time is lost to memory traffic, to the CPU launching thousands of small GPU operations, to padding, and to decoding loops that process one token at a time. The main techniques are batching, kernel fusion, CUDA graphs, reduced precision and careful scheduling of the decoder, and speech adds a constraint most text workloads lack: live audio arrives continuously and cannot be made to wait.
Key takeaways
- A single speech request uses a small fraction of a modern GPU. Batching many requests together is the largest single lever on throughput.
- Small operations are dominated by overhead. Fusing kernels and replaying CUDA graphs cut launch costs and memory round trips.
- The encoder is parallel and efficient; autoregressive or transducer decoding is sequential and often dominates the time.
- Streaming and batch serving want different schedulers. Streaming needs bounded per-chunk latency; batch needs maximum throughput.
- Every optimisation must be checked against accuracy on real audio. Faster and subtly wrong is a regression.
Where the time goes#
A speech model has two phases. The encoder turns audio features, typically log-mel frames every 10 ms downsampled to 40 or 80 ms, into a sequence of vectors. It runs all frames in parallel, using large matrix multiplications that GPUs are designed for. The decoder turns those vectors into text. For CTC it is a cheap parallel step; for transducers and attention encoder-decoders it is a loop, emitting one token or frame decision at a time. See ASR model architectures.
The roofline model (Williams, Waterman and Patterson, 2009) gives the useful frame: an operation is either compute-bound, limited by arithmetic throughput, or memory-bound, limited by how fast data moves from GPU memory. Large encoder matrix multiplications at a good batch size are compute-bound. Decoder steps, normalisation, activations and element-wise operations on small tensors are memory-bound or, worse, dominated by fixed overheads.
| Cost | Where it appears | Typical fix |
|---|---|---|
| Kernel launch overhead | Many small ops per decoder step | CUDA graphs, fewer larger kernels |
| Memory round trips | Chains of element-wise ops, attention | Kernel fusion, fused attention |
| Idle compute | Batch size of one | Dynamic batching |
| Wasted compute | Padding short clips to long ones | Length bucketing, packed sequences |
| Sequential decoding | Autoregressive and transducer loops | Batched decoding, better scheduling |
Batching#
A GPU running one request reads every model weight from memory to process a few seconds of audio. Running 32 requests reads the weights once for all of them. Throughput grows almost linearly with batch size until compute or memory capacity saturates.
Static batching collects a fixed number of inputs and runs them together, which suits offline batch transcription. Padding is the hidden cost: batch a 3-second clip with a 30-second one and the short clip pays for 27 seconds of zeros. Sorting or bucketing inputs by length recovers most of that.
Dynamic batching groups requests that arrive within a short window, a few milliseconds, at serving time. Serving frameworks such as NVIDIA Triton Inference Server implement it generically. For decoders with loops, continuous batching, where sequences join and leave the batch at each step rather than waiting for the slowest one to finish, was described for text generation by Yu et al. (2022) in Orca and applies equally to autoregressive speech decoders.
Kernel fusion and CUDA graphs#
Fusion#
Written naively, a layer such as "add bias, apply activation, add residual, normalise" is four kernels, each reading its input from GPU memory and writing its output back. Fused into one kernel, the data is read once, transformed in registers and on-chip memory, and written once. FlashAttention (Dao et al., 2022) applied the same idea to attention, computing it in tiles so the full attention matrix is never written to memory. Compilers such as TensorRT and torch.compile fuse common patterns automatically; performance-critical engines hand-write kernels for the rest.
CUDA graphs#
Every kernel launch costs the CPU a few microseconds. A decoder step may involve dozens of small kernels, and a transcript hundreds of steps. At small batch sizes the GPU finishes each kernel before the CPU has launched the next, and sits idle. A CUDA graph records the whole sequence once and replays it with a single launch.
Graphs need fixed shapes and memory addresses, which clashes with variable-length audio and changing batch sizes. The usual solution is to capture graphs for a set of batch sizes and sequence-length buckets and pick the closest at runtime.
Precision#
Training often uses mixed precision. Inference commonly runs in 16-bit floating point (FP16 or BF16), which halves memory traffic relative to FP32 and uses tensor cores. 8-bit integer or 8-bit floating point formats, where the hardware supports them, cut it further. Speech models are usually tolerant of 16-bit inference; 8-bit and below needs per-layer care and evaluation on accented, noisy and domain audio, not just a clean test set. See model size and quantisation.
Decoding and scheduling#
Once the encoder is efficient, decoding usually becomes the largest share of the time. Several techniques help:
- Greedy over beam search. For many modern end-to-end models, greedy decoding costs little accuracy and is much cheaper. Beam search multiplies decoder work by the beam width.
- Batched decoder steps. Run one step for every active sequence at once, rather than looping over sequences.
- Separate encoder and decoder scheduling. The encoder can process a large batch of new audio while the decoder serves a different set of in-flight sequences, keeping both busy.
- Keep state on the GPU. Transducer and streaming models carry state between chunks. Moving it to the CPU and back each step adds latency and synchronisation.
Streaming versus batch serving#
Batch transcription can accumulate large batches, sort by length and run at full utilisation; the metric is RTFx and cost per hour of audio. Streaming cannot wait: each stream sends a chunk every few hundred milliseconds, and each chunk must be processed within a bounded time or the latency the user feels grows. A streaming scheduler forms micro-batches from whichever streams have a chunk ready, with a deadline. Serving thousands of streams on a pool of GPUs is covered in concurrency and scaling.
How Reson8 approaches this#
Reson8 serves Resonant-1 with Violin, its own inference engine for speech, running on EU GPU infrastructure. Violin combines fused kernels, CUDA graphs and decoding scheduling designed around speech workloads. On the Open ASR Leaderboard short-form English sets (April 2026), Resonant-1 recorded an RTFx of 1187 at a 6.51% average WER; the benchmarks post has the full tables.
Common questions#
Do I need a data-centre GPU to run speech recognition?
No. Smaller models run on consumer GPUs and CPUs. Data-centre GPUs matter when you need high throughput or many concurrent streams per machine.
Is TensorRT or torch.compile enough?
They capture much of the benefit of fusion and graph capture for standard layers. Custom decoding loops, streaming state and variable-length batching often need extra work beyond what a general compiler provides.
Why is my GPU utilisation low?
Usually small batches, launch overhead or CPU-side work such as audio decoding and resampling starving the GPU. Profile with a tool such as NVIDIA Nsight Systems to see gaps on the timeline.
References#
- Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4).
- Dao, T. et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS.
- Yu, G.-I. et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI.
- Graves, A. (2012). Sequence Transduction with Recurrent Neural Networks. arXiv:1211.3711.
- NVIDIA. CUDA C++ Programming Guide: CUDA Graphs. NVIDIA Docs.
- NVIDIA Triton Inference Server. GitHub.