Domain adaptation is the practice of making a general speech recognition model perform well on a specific kind of speech: a hospital's dictations, a bank's calls, a manufacturer's part numbers. General models are trained to be right on average, so they favour common words and fail on the specialist terms a domain depends on. Adaptation shifts that balance, either by retraining the model on domain data or by steering it at runtime with domain text.
Key takeaways
- A domain differs from general speech in three ways: acoustics (microphones, rooms, accents), vocabulary (names, terms, codes) and style (phrasing, formatting conventions).
- Most practical failures are vocabulary failures. A model that has rarely seen a word will pick a common word that sounds similar.
- Methods split into retraining (fine-tuning on domain audio) and runtime steering (word lists, context, language model fusion), plus post-correction of the text.
- Runtime methods are faster to deploy and easier to update. Retraining can fix acoustic mismatch that text cannot.
- Measure adaptation on the target terms and on ordinary words. Gains on the first can come at the cost of the second.
What makes a domain#
A "domain" is any setting where speech differs systematically from what a model saw in training. The differences fall into three groups.
- Acoustic. Channel, noise and speakers. Telephone audio at 8 kHz, a car cabin, a consultation room with a ceiling microphone, a population with particular accents.
- Lexical. The words themselves. Drug names, legal terms, company and product names, staff names, street names, and identifiers such as order codes.
- Stylistic. How things are said and written. Clinicians dictate in a compressed telegraphic style; traders use shorthand; each organisation has its own conventions for numbers and abbreviations.
Modern general models trained on hundreds of thousands of hours are far more tolerant of acoustic variety than their predecessors. The remaining gap is mostly lexical. That is why domain adaptation today is largely about vocabulary and context.
Why general models fail on specialist speech#
A speech model chooses the most probable text given the audio. That probability combines what the audio sounds like with what the model has learned is likely to be said. For a common phrase, both agree. For a rare term, the acoustic evidence points one way and the learned prior points to a familiar word that sounds similar, and the prior often wins.
| Domain | Said | Heard by a general model |
|---|---|---|
| Healthcare | metformin | met for min |
| Healthcare | apixaban | a pixie ban |
| Finance | EBITDA | a bit da |
| Logistics | INV7625X | INB seven six two five X |
| Local names | Scheveningen | Schaven ingen |
These are not random errors. They are the model doing what it was trained to do. Each rare word is individually infrequent, but in a specialist domain the long tail of such words carries much of the information. A transcript can be 95% correct and miss every term a reader needed.
The main methods#
Fine-tuning on domain audio#
Continue training the model on transcribed audio from the domain, updating all weights or a small set of added ones such as adapters or low-rank (LoRA) matrices. This is the only method that fixes acoustic mismatch directly. It needs tens to hundreds of hours of transcribed audio, a training pipeline, and care to avoid degrading general accuracy. Every vocabulary change means another training run.
Runtime biasing with text#
Leave the model's weights alone and give it domain information at inference time. The simplest form is a word list whose entries receive a score boost during decoding. Richer forms pass free text, such as a patient's medication list or a meeting agenda, to a model that can attend to it, as in contextual listen-attend-spell (CLAS) and later contextual models. Long-context biasing extends this to entire documents. See fine-tuning vs context biasing for the trade-offs.
Language model fusion#
Train a language model on domain text and combine its scores with the speech model's during decoding. Shallow fusion does this by weighted addition. Refinements such as internal language model estimation subtract the speech model's own built-in prior first, so the domain model has more influence. This is a form of text-only domain adaptation: it needs text, not audio.
Post-correction#
Fix the transcript after recognition, with replacement rules or a language model that rewrites likely errors. It is easy to add, but it works only on the text, so it cannot recover a word whose sound was lost, and a language model rewriting transcripts can introduce plausible errors of its own.
Choosing an approach#
| Situation | Usually best served by |
|---|---|
| Vocabulary changes weekly (products, staff, patients) | Runtime biasing |
| Thousands of specialist terms, stable | Long-context biasing or LM fusion |
| Unusual acoustics, e.g. a new microphone type or language variety | Fine-tuning |
| Fixed identifier formats | Format-aware biasing and normalisation |
| No audio can be collected for privacy reasons | Text-only methods |
Methods combine. A fine-tuned model can still take runtime context, and every approach benefits from good formatting rules.
How Reson8 approaches this#
Reson8's domain adaptation is text-only. Domain text is supplied through the API, with no fine-tuning, audio data or retraining, and takes effect in under 60 seconds. Up to 1M tokens of context can be applied at runtime without latency cost, including 50,000+ medical terms. For identifiers with a known format, Patterns biases the model towards an expected shape such as INV[0-9]{3}. Like other text-only methods, it acts on vocabulary and context rather than on acoustics. Details are in the documentation.
Common questions#
Is domain adaptation the same as fine-tuning?
No. Fine-tuning is one method of domain adaptation. Runtime biasing, language model fusion and post-correction are others, and they do not change the model's weights.
How much data do I need?
For runtime biasing, a list of terms or a set of representative documents. For fine-tuning, typically tens to hundreds of hours of transcribed audio, depending on how far the domain is from the model's training data.
Can adaptation make things worse?
Yes. Aggressive biasing can make the model write domain terms where ordinary words were spoken, a problem called over-biasing. Fine-tuning on narrow data can reduce accuracy on general speech. Always measure both.
Do large general models still need adaptation?
Less for acoustics than older models did, but still for vocabulary. No training set contains every product name, colleague and patient, and those are the words users care about most.
References#
- Bell, P., Fainberg, J., Klejch, O., Li, J., Renals, S., & Swietojanski, P. (2021). Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview. IEEE Open Journal of Signal Processing.
- Leggetter, C. J., & Woodland, P. C. (1995). Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models. Computer Speech & Language, 9(2).
- Pundak, G., Sainath, T. N., Prabhavalkar, R., Kannan, A., & Zhao, D. (2018). Deep Context: End-to-End Contextual Speech Recognition. IEEE SLT.
- Toshniwal, S. et al. (2018). A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition. IEEE SLT.
- Meng, Z. et al. (2021). Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition. IEEE SLT.
- Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.