Custom vocabulary and context biasing tell a speech recognition system which words to expect, so that it chooses a rare term over a common one that sounds similar. The simplest form is a list of words with a boost weight; richer forms pass phrases, documents or expected formats alongside the audio. It is the fastest way to fix errors on names, products and technical terms, and it needs no audio or retraining. Used carelessly, it also inserts listed terms where they were never said.
Key takeaways
- Biasing works by adding a bonus to listed terms during decoding. It helps when the audio evidence is ambiguous, not when the word was not audible.
- Short, targeted lists work better than long generic ones with simple boosting. Every unspoken term is a chance of a false insertion.
- Per-request lists, built from what is known about the call or document, beat one global list.
- Write terms as they should appear in the transcript, include multi-word phrases whole, and add common inflected forms where the language needs them.
- Measure recall on listed terms, false insertions, and WER on unlisted words. All three should improve or hold.
How biasing works#
A speech recogniser keeps several candidate transcripts during beam search and scores each by how well it fits the audio and how likely the words are. For a rare term, a common phrase that sounds alike often scores slightly higher because the model has seen it far more often. Biasing adds a bonus to candidates containing listed terms, enough to tip close contests.
Implementations#
- Score boosting in search. Listed terms are stored in a prefix tree. As decoding extends a candidate through a term's prefix, the bonus accumulates; if the candidate leaves the term before completing it, the partial bonus is typically removed. This is the classic "shallow fusion" approach and is widely used in production systems.
- Class-based language models. Older hybrid systems placed slot classes such as
$CONTACTor$SONGin a language model and filled them at runtime with a user's own list. Google described on-the-fly rescoring of this kind for contacts and apps in 2015. - Contextual encoders. The model is trained with an extra input: a set of context phrases it can attend to while decoding. CLAS introduced this for attention-based models, and variants exist for transducers.
- Prompts and long context. Models with a text decoder can be conditioned on free text, from a short prompt to whole documents. See long-context biasing.
Commercial APIs expose these under different names. Amazon Transcribe calls them custom vocabularies; Google Cloud Speech-to-Text calls its version speech adaptation, with phrase sets and boost values. The underlying idea is the same.
Writing a good list#
Most of the benefit comes from what goes in the list, not from tuning weights.
- Include what the model gets wrong. Run your test set without biasing and collect the terms it misses. Common words it already handles add risk and no benefit.
- Write the output form. Enter Scheveningen, apixaban or EBITDA as they should appear. Some systems accept a "sounds like" hint for unusual pronunciations; use it for brand names that are not pronounced as spelt.
- Keep phrases whole. Kaiser Permanente as one entry is stronger evidence than two separate words, and less likely to fire on "Kaiser" alone.
- Cover inflection. In Polish, German or Dutch, a name or term can take several forms. List the ones that occur, or use a system that matches forms.
- Scope it to the request. The caller's own products, the patient's own medications, the agenda for this meeting. A small, relevant list outperforms a large, generic one with simple boosting.
| Entry | Problem | Better |
|---|---|---|
| order | Common word, already recognised; may distort nearby text | Leave out |
| Kaiser | Fires on unrelated uses | Kaiser Permanente |
| EBIDTA | Misspelt; the typo will appear in transcripts | EBITDA |
| all 5,000 SKUs | Thousands of distractors per call | SKUs on this customer's account |
Identifiers and formats#
Word lists do not help much with order numbers, licence plates or policy numbers, because the set of possible values is too large to list. What is known is the shape: three letters and four digits, a country's plate format. Systems that accept a format, rather than a list, can bias towards any value of that shape. The speech is also different: callers spell codes letter by letter, say "double seven", and mix digits with phonetic alphabet words. See structured identifiers and numbers and alphanumerics.
Measuring the effect#
Score three things, with and without biasing, on the same test set:
- Recall on listed terms. Did more of them come out right? See entity and keyword accuracy.
- False insertions. How often does a listed term appear where it was not said?
- WER on unlisted words. Has the rest of the transcript got worse?
Test with realistic lists that include distractors: terms that are relevant to the domain but not spoken in a given clip. A test where every listed term is spoken shows only the upside. When false insertions rise faster than recall, the system is over-biasing.
How Reson8 approaches this#
Reson8's domain adaptation takes text rather than a short word list: domain text is supplied through the API and takes effect in under 60 seconds, with up to 1M tokens of context applied at runtime and no added latency, including 50,000+ medical terms. No audio or retraining is involved. For identifiers, Patterns accepts an expected format such as INV[0-9]{4}[A-Z]; a code that was transcribed as INB seven six two five X comes out as INV7625X. On Reson8's tests, Patterns cut order-number WER from 8.0% to 1.5% and licence-plate WER from 6.6% to 4.7%.
Common questions#
How many terms can a list hold?
It depends on the method. Simple boosting often works best with tens to a few hundred terms per request, because each term adds risk of false matches. Methods built for long context accept far more. Check the vendor's limits and test at your real size.
Can biasing fix a word the model cannot hear?
No. If the audio is too noisy or the word was cut off, no bonus will recover it. Biasing only helps when the right answer was already a close candidate.
Is custom vocabulary the same as fine-tuning?
No. Custom vocabulary steers a fixed model at runtime; fine-tuning changes the model's weights through training. See fine-tuning vs context biasing.
Should I add pronunciations?
Only where the system supports them and the spelling is misleading, as with some brand names and foreign names. For most terms, the written form is enough for modern models.
References#
- Aleksic, P. et al. (2015). Bringing Contextual Information to Google Speech Recognition. Interspeech.
- Hall, K. et al. (2015). Composition-based on-the-fly rescoring for salient n-gram biasing. Interspeech.
- Pundak, G., Sainath, T. N., Prabhavalkar, R., Kannan, A., & Zhao, D. (2018). Deep Context: End-to-End Contextual Speech Recognition. IEEE SLT.
- Jain, M. et al. (2020). Contextual RNN-T for Open Domain ASR. Interspeech.
- Le, D. et al. (2021). Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion. Interspeech.
- Amazon Transcribe: Custom vocabularies. AWS documentation.