Acoustic models and language models

Classical speech recognition split the problem in two. An acoustic model scored how well the audio matched candidate sounds, and a language model scored how plausible each candidate word sequence was, and a decoder searched for the sequence that maximised both. End-to-end neural models merged the two into one network trained on audio with transcripts. The split still matters, because a language model can learn from text alone, and text is far easier to get than transcribed audio.

The noisy channel formulation#

From the 1970s onward, speech recognition was framed as decoding a message sent through a noisy channel. The speaker intends a word sequence W; the channel, meaning the vocal tract, the room and the microphone, turns it into acoustic observations X. The recogniser inverts the channel by finding the most probable W. By Bayes' rule, and dropping P(X) because it is the same for every candidate:

P(X | W) is the acoustic model: how likely this audio is if these words were spoken. P(W) is the language model: how likely these words are in the first place. In practice the language model score is raised to a tuned weight and combined with a word insertion penalty, because the two models are estimated on different scales.

The split is useful because the two halves learn from different data. The acoustic model needs audio aligned to transcripts, which is expensive. The language model needs only text, which exists in vast quantities for most languages and domains.

The classical pipeline#

Acoustic models#

In hybrid systems the acoustic model scored short stretches of audio against sub-word sound units. Words were modelled as sequences of phones, and each phone, in the context of its neighbours (a triphone), as a small hidden Markov model with three or so states. For each state, a Gaussian mixture model gave the probability of the observed features. From around 2010, a deep neural network replaced the Gaussian mixtures, predicting the HMM state for each frame; this "hybrid DNN-HMM" design delivered the first large deep-learning gains in ASR.

The lexicon#

A pronunciation dictionary linked the two halves. "Tomato" might have two entries with different vowels. Words missing from the lexicon could not be recognised at all, which made adding vocabulary an explicit and sometimes laborious task, but also a predictable one: add the word and its pronunciation, and the system could output it.

Language models#

The language model was almost always an n-gram model: the probability of each word given the previous two to four, estimated by counting in a text corpus and smoothed so that unseen sequences did not get zero probability. Kneser-Ney smoothing, introduced in 1995, became the standard. Language model quality was measured by perplexity on held-out text. Acoustic model, lexicon and language model were compiled into a single weighted finite-state transducer and searched with beam pruning. Larger neural language models were often used in a second pass to rescore the top hypotheses.

How the halves merged#

End-to-end models, whether CTC, transducer or attention encoder-decoder, map features directly to sub-word tokens. There is no lexicon, no explicit phone set, and no separate language model. The network learns acoustic patterns and the regularities of the language jointly from paired audio and transcripts.

The decoder in a transducer or attention model therefore contains an internal language model. It works well for everyday language. Its limit is the training transcripts: a few hundred thousand hours of audio is on the order of a few billion words of text, far less than the text a modern language model sees, and most of it is general conversation, broadcast and read speech. Domain terms that rarely appear in speech corpora are underrepresented, so the model substitutes a common word that sounds similar: tachycardic becomes taking a cardic.

Bringing language models back#

Several techniques add external language knowledge to end-to-end models:

• N-best rescoring. Generate several candidate transcripts, then re-rank them with a stronger language model. Simple, but limited to candidates the first pass produced.
• Shallow fusion. Add a weighted external LM score at every beam search step. Widely used and requires no retraining of the speech model.
• Internal LM estimation and density ratio methods. Subtract an estimate of the internal LM before adding the external one, so domain text replaces the general prior rather than stacking on it.
• Deep and cold fusion. Integrate the LM's hidden states into the decoder during training. More powerful, but the speech model must be retrained.
• Context biasing. Boost specific words or phrases during search. See custom vocabulary and context biasing.

Shallow fusion in one line: λ is tuned on a development set. Too low and the LM has no effect; too high and the output drifts toward what the LM expects instead of what was said, the same risk as over-biasing.

Large language models as decoders#

A more recent pattern connects a speech encoder to a pre-trained large language model through an adapter, so the LLM acts as the decoder. This inherits the LLM's broad text knowledge and its ability to follow instructions and use long context, which is why prompting such systems with domain text can improve recognition of rare terms. It also inherits LLM costs and failure modes, including slower autoregressive decoding and a greater tendency to produce fluent text that the audio does not support.

How Reson8 approaches this#

Reson8's domain adaptation uses the old insight that text is cheaper than audio. Customers supply domain text through the API, up to 1M tokens of context, and it is applied at runtime without fine-tuning, audio data or retraining, taking effect in under 60 seconds and without latency cost. The mechanism is described in text-only domain adaptation and long-context biasing.

Common questions#

References#

1. Jelinek, F. (1976). Continuous speech recognition by statistical methods. Proceedings of the IEEE, 64(4).
2. Kneser, R., & Ney, H. (1995). Improved backing-off for m-gram language modeling. ICASSP.
3. Mohri, M., Pereira, F., & Riley, M. (2002). Weighted finite-state transducers in speech recognition. Computer Speech & Language, 16(1).
4. Hinton, G. et al. (2012). Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6).
5. Kannan, A. et al. (2018). An analysis of incorporating an external language model into a sequence-to-sequence model. ICASSP.
6. Gulcehre, C. et al. (2015). On using monolingual corpora in neural machine translation. arXiv:1503.03535.
7. Meng, Z. et al. (2021). Internal language model estimation for domain-adaptive end-to-end speech recognition. IEEE SLT.