Speech recognition has moved through four broad eras in about seventy years. Systems in the 1950s and 1960s matched sounds against stored templates and recognised a few words from one speaker. From the 1970s, statistical models built on hidden Markov models took over and dominated for three decades. Deep neural networks replaced the acoustic core around 2010 to 2012, and since the mid-2010s end-to-end neural models trained on vast amounts of audio have replaced the rest of the pipeline.
Key takeaways
- The first recognisers, such as Bell Labs' Audrey in 1952, recognised spoken digits from a single speaker using hand-built circuitry.
- The decisive shift of the 1970s and 1980s was from linguistic rules to statistics: learn probabilities from data, then search for the most likely words.
- HMM systems with Gaussian mixtures and n-gram language models were the standard design from the 1980s until around 2012.
- Deep neural networks first replaced the Gaussian mixtures, then end-to-end models such as CTC, RNN-T and attention encoder-decoders replaced the whole pipeline.
- Progress has tracked data and compute as much as algorithms: shared corpora, public benchmarks and GPUs each mark a turning point.
Timeline#
1950s to 1960s: circuits and templates#
In 1952, K. H. Davis, R. Biddulph and S. Balashek at Bell Laboratories described a system, later known as Audrey, that recognised the spoken digits zero to nine. It measured how energy was distributed across two frequency bands over time and compared the result to stored patterns for each digit. It worked well for the speaker it was tuned to and poorly for anyone else.
IBM showed its Shoebox machine at the 1962 Seattle World's Fair. It recognised sixteen words, the digits plus arithmetic commands, and could do simple sums on request. Laboratories in the United States, the United Kingdom, Japan and the Soviet Union built similar isolated-word systems throughout the decade.
The key technical idea of the era was template matching with time alignment. A word spoken slowly and the same word spoken quickly produce different-length feature sequences. Dynamic time warping, developed by Vintsyuk in the Soviet Union in the late 1960s and refined by Sakoe and Chiba in Japan in 1978, aligns them with dynamic programming. DTW remained useful for small-vocabulary, speaker-dependent tasks for years.
1970s to 2000s: the statistical turn#
From 1971 to 1976, the US Advanced Research Projects Agency funded the Speech Understanding Research programme, which asked for continuous speech with a vocabulary of about a thousand words. Carnegie Mellon's Harpy met the target with 1,011 words, using a precompiled network of all allowed sentences and beam search, a technique still used in decoders today.
The lasting change came from statistical modelling. James Baker's Dragon system at Carnegie Mellon and Frederick Jelinek's group at IBM modelled speech as a hidden Markov model: a sequence of hidden states, one per sound unit, each producing acoustic features with learned probabilities. Recognition became a search for the most probable word sequence, combining an acoustic model, a pronunciation dictionary and an n-gram language model. Parameters were estimated from data rather than written by linguists.
Through the 1980s and 1990s this design was refined rather than replaced. Gaussian mixture models described each state's features. Context-dependent triphones captured how a sound changes with its neighbours. Speaker adaptation, discriminative training and weighted finite-state transducers, formalised by Mohri, Pereira and Riley, made systems more accurate and decoding more efficient. Kai-Fu Lee's Sphinx at CMU showed speaker-independent continuous recognition in the late 1980s.
Shared data drove the work. The TIMIT corpus, the Wall Street Journal read-speech corpus and the Switchboard telephone conversation corpus gave researchers common test sets, and US government evaluations tracked word error rate year by year. Commercial dictation followed: Dragon NaturallySpeaking, released in 1997, was among the first consumer products to accept continuous speech rather than words separated by pauses.
2009 to 2015: deep learning enters#
Neural networks had been tried for speech since the late 1980s, including hybrid HMM and neural network systems, but they did not clearly beat well-tuned Gaussian mixtures. That changed around 2009 to 2012, when deeper networks, better initialisation and GPU training made it practical to replace the GMMs with a deep neural network that predicted HMM states from a window of frames. A 2012 paper by Hinton and colleagues from Toronto, Microsoft, Google and IBM reported large error reductions across several tasks, and industry adopted the approach within a few years.
The rest of the classical pipeline stayed in place for a while: HMM topology, pronunciation lexicon, n-gram language model and WFST decoding. The open-source Kaldi toolkit, released in 2011, became the standard research platform for these hybrid systems.
2014 onwards: end-to-end models#
The next step removed the hand-designed parts. Three families of end-to-end architectures emerged:
- CTC (connectionist temporal classification), introduced by Graves and colleagues in 2006, lets a network emit characters directly without a frame-level alignment. Baidu's Deep Speech systems in 2014 and 2015 showed it at scale.
- RNN-T (recurrent neural network transducer), proposed by Graves in 2012, adds a prediction network and suits streaming. Google used it for on-device recognition on phones.
- Attention encoder-decoders, such as Listen, Attend and Spell in 2016, generate text token by token while attending over the whole utterance.
The Transformer, introduced for machine translation in 2017, and the Conformer, which added convolution to it in 2020, became the dominant encoders. In 2016 Microsoft reported word error rates on the Switchboard conversational benchmark comparable to professional human transcribers, a milestone that also sparked debate about how human performance should be measured.
Scale and self-supervision#
From 2020, the bottleneck moved from architecture to data. wav2vec 2.0 showed that a model pre-trained on unlabelled audio could be fine-tuned with very little transcribed speech. OpenAI's Whisper, released in 2022, was trained on 680,000 hours of weakly labelled audio from the web and generalised to many domains and languages without fine-tuning. Current systems combine large encoders, sub-word decoders, heavy data curation and, increasingly, training methods borrowed from large language models, including reinforcement learning.
What the history shows#
Three patterns recur. Data beats rules: each time more data became available, approaches that learned from it overtook approaches built on expert knowledge. Benchmarks shape progress: shared test sets made improvement measurable, but also encouraged tuning to them, a problem that persists as benchmark contamination. And the hard cases have barely changed: conversational speech, noise, accents and rare words were the open problems in the 1990s and remain the places where error rates are highest.
How Reson8 approaches this#
Reson8's model, Resonant-1, belongs to the latest era: an end-to-end neural model trained on a four-phase schedule that relies heavily on reinforcement learning, served on its own inference engine, Violin. Reson8 excludes LibriSpeech and VoxPopuli from its reported benchmark averages because they were used in training, a direct response to the contamination problem that long-running shared benchmarks create.
Common questions#
What was the first speech recognition system?
Bell Labs' 1952 digit recogniser, usually called Audrey, is generally cited as the first. It recognised spoken digits from a single speaker.
When did speech recognition become usable for ordinary people?
Consumer dictation software with continuous speech appeared in the late 1990s, but it needed training on the user's voice and a quiet room. Speaker-independent recognition good enough for everyday use on phones arrived in the early 2010s with deep neural networks and cloud processing.
Are hidden Markov models still used?
Rarely in new large systems, which are end-to-end. HMM-based hybrid systems still run in some production deployments and embedded devices, and ideas from that era, such as beam search and WFSTs, survive inside modern decoders.
References#
- Davis, K. H., Biddulph, R., & Balashek, S. (1952). Automatic recognition of spoken digits. Journal of the Acoustical Society of America, 24(6).
- Sakoe, H., & Chiba, S. (1978). Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 26(1).
- Jelinek, F. (1976). Continuous speech recognition by statistical methods. Proceedings of the IEEE, 64(4).
- Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2).
- Juang, B. H., & Rabiner, L. R. (2005). Automatic speech recognition: a brief history of the technology development. Elsevier Encyclopedia of Language and Linguistics.
- Hinton, G. et al. (2012). Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6).
- Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006). Connectionist temporal classification. ICML.
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.