The size of a speech model, measured in parameters, and the precision of the numbers it computes with together decide how much memory it needs, how fast it runs and where it can be deployed. Making a model smaller, by designing it smaller, distilling it from a larger one, pruning it, or quantising its weights from 32-bit floats to 8 or 4 bits, reduces cost and enables on-device use. The price is usually paid in accuracy, and it is rarely paid evenly: rare words, accented speech and noisy audio degrade first, while clean benchmark scores barely move.
Key takeaways
- Memory for weights is roughly parameters × bytes per parameter. A 1-billion-parameter model needs about 4 GB in FP32, 2 GB in FP16 and 1 GB in INT8.
- 16-bit inference is nearly free in accuracy for most speech models. 8-bit usually works with care. 4-bit and below needs careful evaluation.
- Quantisation reduces memory traffic more reliably than it reduces latency. Speed-ups depend on hardware support for the format.
- Distillation shrinks the architecture itself and often preserves more accuracy than quantising a large model aggressively.
- Evaluate compressed models on hard audio, not only on the clean test set. Losses concentrate in the long tail.
What model size means#
Speech recognition models range from a few million parameters, for on-device command recognition, to well over a billion for general-purpose multilingual transcription. Whisper illustrates the spread within one family:
| Whisper model | Parameters | Approx. FP16 weights |
|---|---|---|
| tiny | 39 M | ~80 MB |
| base | 74 M | ~150 MB |
| small | 244 M | ~0.5 GB |
| medium | 769 M | ~1.5 GB |
| large | 1,550 M | ~3.1 GB |
Parameter counts are from Radford et al. (2022); weight sizes are computed at two bytes per parameter. The paper reports accuracy improving with model size across its evaluations.
Size is not the only factor in speed. Where parameters sit matters. In an encoder-decoder model, a large encoder runs once per input in parallel, while each decoder layer runs once per output token. Distil-Whisper (Gandhi et al., 2023) exploited this by keeping Whisper's full encoder and cutting the decoder to two layers, producing a model several times faster with a small loss in accuracy on out-of-distribution test sets. See ASR model architectures.
Number formats#
Each parameter is stored in some number format. Fewer bits mean less memory and less data to move per operation.
- FP32: the default for training historically, rarely needed for inference.
- FP16 and BF16: 16-bit floats. BF16 keeps FP32's exponent range with less precision, which avoids overflow problems. Standard for GPU inference.
- INT8 and FP8: 8-bit formats. INT8 is widely supported on CPUs and GPUs; FP8 is supported on recent data-centre GPUs.
- INT4 and lower: common for large language models on consumer hardware, and used by on-device speech runtimes such as whisper.cpp, which offers several quantised formats.
How quantisation works#
Integer quantisation maps a range of real values to a small set of integers using a scale and, optionally, a zero point:
The scale s can be set per tensor, per output channel or per small group of weights. Finer granularity preserves accuracy better at a small storage cost. Jacob et al. (2018) described the integer-arithmetic scheme that became the basis of many mobile inference runtimes.
Weights only, or weights and activations#
Weight-only quantisation stores weights in low precision and converts them back to 16-bit just before each multiplication. It cuts memory and memory traffic, which helps memory-bound workloads such as small-batch decoding, but the arithmetic is still 16-bit. Weight and activation quantisation also runs the arithmetic in 8-bit, which can speed up compute-bound work, but activations have outliers that make it harder to do without accuracy loss. Dettmers et al. (2022) showed that a small number of outlier features dominate this problem in large transformers.
Post-training versus quantisation-aware training#
Post-training quantisation takes a trained model and a small calibration set, measures value ranges, and picks scales. It is quick and needs no training pipeline. Methods such as GPTQ (Frantar et al., 2023) use calibration data to adjust remaining weights and compensate for rounding error. Quantisation-aware training simulates low precision during training or fine-tuning so the model learns to tolerate it, and usually recovers more accuracy at 4 bits and below.
Distillation and pruning#
Knowledge distillation (Hinton, Vinyals and Dean, 2015) trains a small student model to match a large teacher's outputs. In speech it is often done with pseudo-labels: the teacher transcribes a large amount of unlabelled audio, and the student learns from those transcripts. The student has fewer layers or narrower ones and is fast at full precision, and it can then be quantised as well.
Pruning removes weights, attention heads or whole layers that contribute little. Unstructured pruning, which zeroes individual weights, rarely speeds up inference without specialised hardware support. Structured pruning, which removes whole units, gives real speed-ups but usually needs fine-tuning afterwards.
What compression costs#
Compression errors are not uniform. A compressed model often keeps its WER on a clean benchmark within a fraction of a point, while losing more on:
- rare words and proper nouns, which depend on fine distinctions in the output distribution;
- accented and noisy speech, which exercises the edges of the activation ranges;
- languages with less training data, in multilingual models;
- hallucination behaviour on silence and music, which can change after quantisation.
Evaluate the compressed model against the original on a test set built from your own audio, broken down by condition, and look at entity accuracy as well as aggregate WER. See building your own ASR test set and evaluation beyond WER.
Choosing a size and precision#
| Deployment | Typical choice |
|---|---|
| Microcontroller wake word | Tiny model, INT8 |
| Phone or laptop dictation | Small to medium model, 8-bit or 4-bit weights |
| Cloud streaming at scale | Largest model the latency budget allows, FP16 or BF16, 8-bit where validated |
| Cloud batch transcription | Accuracy-first model, 16-bit, large batches |
In the cloud, a larger model at high utilisation often costs less per hour of audio than expected, because batching spreads its cost. On a device, memory and battery set hard limits. See on-device vs cloud speech recognition.
Common questions#
Does quantisation always make a model faster?
No. It reduces memory, and often speed, but only when the hardware and runtime have efficient kernels for the format. Some formats are slower than FP16 on hardware without native support.
Is a quantised large model better than a full-precision small one?
Often, at moderate compression such as 8-bit. At very low bit widths a well-distilled smaller model can be the better choice. Test both on your audio.
How much accuracy does INT8 cost?
For many speech models, little on clean audio when done carefully. The only reliable answer is to measure on your own hard cases.
References#
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.
- Gandhi, S., von Platen, P., & Rush, A. M. (2023). Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling. arXiv:2311.00430.
- Jacob, B. et al. (2018). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR.
- Dettmers, T. et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS.
- Frantar, E. et al. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR.
- Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
- whisper.cpp. GitHub.