Fine-tuning and context biasing are the two main ways to adapt a speech recognition model to a domain. Fine-tuning retrains the model on transcribed domain audio, so its weights change permanently. Context biasing leaves the model as it is and supplies domain words or text with each request, steering decoding at runtime. Fine-tuning can fix acoustic mismatch; context biasing is faster, cheaper to update and needs no audio. For most vocabulary problems, biasing is the better first step.
Key takeaways
- Fine-tuning needs transcribed domain audio, a training run and a fresh evaluation for every change. Context biasing needs text and takes effect immediately.
- Only fine-tuning addresses acoustic mismatch, such as an unusual microphone or an under-represented accent.
- Context biasing can differ per request: one patient's medication list, one caller's account details. A fine-tuned model is the same for everyone who uses it.
- Both can hurt general accuracy. Fine-tuning through forgetting; biasing through over-biasing. Measure ordinary words as well as target terms.
- They combine. A fine-tuned model can still accept runtime context.
Two ways to change behaviour#
A speech model maps audio to text using what it learned in training. To make it better at a domain, you can change what it has learned, or change what it is told at the moment it transcribes. The first is fine-tuning. The second is context biasing, sometimes called contextual biasing, prompting or custom vocabulary, depending on the form it takes.
How fine-tuning works#
Take a pretrained model and continue training on pairs of domain audio and reference transcripts. Full fine-tuning updates every weight. Parameter-efficient methods freeze the model and train a small set of added parameters: adapter layers, or low-rank update matrices as in LoRA. These are cheaper to train and store, and several domain adapters can share one base model.
The requirements are substantial. You need transcribed audio that represents the domain, usually tens of hours at least, with references produced to a consistent standard. You need a training pipeline, a held-out test set, and a way to deploy and version the result. Each organisation or domain may end up with its own model, which must be retested whenever the base model is upgraded.
What can go wrong#
- Forgetting. A network trained further on narrow data can lose ability it had before. McCloskey and Cohen described this "catastrophic interference" in 1989; it remains the main risk of fine-tuning on a small domain set. General accuracy, and sometimes other languages, degrade.
- Overfitting. With little data, the model learns the particular speakers and sentences rather than the domain.
- Stale vocabulary. A word that did not appear in the fine-tuning data is no better recognised than before. New products and new names need a new training run.
- Data governance. The audio used for training is personal data. Collecting, storing and training on it needs a lawful basis and controls. See GDPR and voice data.
How context biasing works#
The model's weights stay fixed. At inference time it receives extra information about what is likely to be said, and decoding is nudged towards it. The main forms are:
- Boosted word lists. Terms in the list receive a score bonus during beam search, often implemented with a prefix tree so partial matches are rewarded as decoding proceeds. Le and colleagues describe a trie-based approach combined with shallow fusion for streaming models.
- Contextual encoders. The model is trained to attend to a set of context phrases supplied with the audio. Google's contextual listen-attend-spell (CLAS) was an early example.
- Prompting. Models with a text decoder can be conditioned on preceding text. Whisper, for instance, accepts a prompt of prior context, limited to a couple of hundred tokens.
- Long context. Models built for it can take whole documents: a formulary, a case file, a product catalogue. See long-context biasing.
Because the context travels with the request, it can be personal to that request. A clinic can send the current patient's medications; a contact centre can send the caller's account and product names. No fine-tuned model can do that.
Biasing has its own failure mode. Push too hard and the model writes list terms where ordinary words were spoken: met for lunch becomes metformin lunch. This is over-biasing, and it gets worse as lists grow if the method does not scale well.
Side by side#
| Fine-tuning | Context biasing | |
|---|---|---|
| Input needed | Transcribed domain audio | Domain text or term lists |
| Time to take effect | Days to weeks | Immediate to minutes |
| Updating vocabulary | New training run | Edit the text |
| Per-request personalisation | No | Yes |
| Fixes acoustic mismatch | Yes | No |
| Main risk | Forgetting, overfitting | Over-biasing |
| Models to maintain | One per domain or customer | One shared model |
| Privacy footprint | Audio stored for training | Text sent at request time |
Which to use#
Start with context biasing when the problem is vocabulary: names, terms, codes. It is faster to try, easier to reverse, and avoids collecting audio. In modern systems, many domain errors are of this kind; see what is domain adaptation.
Consider fine-tuning when errors persist on ordinary words in your audio, which points to acoustics: a particular device, a noisy environment, a regional variety or a language the base model handles poorly. Also consider it when you have large volumes of well-transcribed domain audio already, and the vocabulary is stable.
The two are not exclusive. A model fine-tuned for a telephone channel can still take a per-call word list. What matters is that each change is measured against a fixed test set, on target terms and on everything else.
How Reson8 approaches this#
Reson8 takes the context route. Its domain adaptation is text-only: domain text is supplied through the API with no fine-tuning, audio data or retraining, and takes effect in under 60 seconds. It accepts up to 1M tokens of context, applied at runtime without latency cost. For identifiers with known formats, Patterns biases the model toward an expected shape, which cut order-number WER from 8.0% to 1.5%.
Common questions#
Is prompting a model the same as context biasing?
It is one form of it. Prompting passes text through the model's decoder as preceding context. Other forms add score bonuses during search or use a dedicated context encoder. All leave the weights unchanged.
How much audio does fine-tuning need?
It depends on how far the domain is from the base model and on the method. Parameter-efficient methods can show gains with modest amounts, but reliable, general gains usually need tens of hours or more of well-transcribed audio.
Can context biasing handle thousands of terms?
Depends on the method. Simple boosting degrades as lists grow, because each added term is another chance for a false match. Methods designed for long context scale further. Test with realistic list sizes.
Does fine-tuning on my data mean the vendor keeps my audio?
Usually, for at least the duration of training, and the resulting model encodes information from it. Check retention and ownership terms in the contract before sending audio for fine-tuning.
References#
- McCloskey, M., & Cohen, N. J. (1989). Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of Learning and Motivation, 24.
- Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.
- Pundak, G., Sainath, T. N., Prabhavalkar, R., Kannan, A., & Zhao, D. (2018). Deep Context: End-to-End Contextual Speech Recognition. IEEE SLT.
- Le, D. et al. (2021). Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion. Interspeech.
- Bell, P. et al. (2021). Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview. IEEE Open Journal of Signal Processing.
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.