Modern speech recognisers share a neural encoder and differ mainly in how they turn its output into text. CTC predicts a token or blank for each frame independently and is the fastest. Transducers such as RNN-T add a small network that remembers what has been emitted, and stream naturally. Attention encoder-decoders generate text one token at a time over the whole utterance, which gives strong accuracy on complete recordings but makes streaming harder and allows hallucination.
Key takeaways
- The encoder, now usually a Conformer or Transformer, does most of the modelling. The decoder family determines speed, streaming behaviour and failure modes.
- CTC is non-autoregressive: one parallel pass, no memory of previous outputs. Fast and stable, weaker on fluency without an external language model.
- Transducers are the usual choice for low-latency streaming. They emit tokens as audio arrives and cannot run ahead of it.
- Attention decoders, as in Whisper, are autoregressive and see the whole input. They are accurate but can loop or invent text on silence.
- Hybrids are common: a shared encoder trained with CTC and attention together, or a transducer that also predicts durations.
The three families#
CTC#
Connectionist temporal classification, introduced by Graves and colleagues in 2006, solves the alignment problem: the audio has hundreds of frames, the transcript a few dozen tokens, and nobody has marked which frame belongs to which token. CTC adds a blank symbol and lets the network output one symbol per encoder frame. At inference, consecutive repeats are merged and blanks removed, so _ _ h h _ e _ l l _ l o _ becomes "hello". During training, the CTC loss sums the probability of every frame sequence that collapses to the correct transcript.
The defining property is conditional independence: each frame's prediction depends on the audio but not on the other predictions. That makes decoding a single parallel pass, which is very fast on GPUs. It also means the model cannot check that its output reads as fluent text, so pure CTC systems benefit more than others from an external language model or beam search with lexical constraints. Well-known CTC systems include wav2vec 2.0 fine-tuned models and NVIDIA's Parakeet CTC models.
Transducers (RNN-T and variants)#
The recurrent neural network transducer, proposed by Graves in 2012, keeps CTC's blank but adds a prediction network that reads the tokens emitted so far, much like a small language model. A joiner combines the encoder's current frame with the predictor's state and decides whether to emit a token or advance to the next frame. The name survives even though the predictor and encoder are no longer usually recurrent; Conformer encoders and stateless predictors are common.
Transducers are a natural fit for streaming. Output is tied to frames already processed, so the model cannot get ahead of the audio, and with a causal or limited-lookahead encoder it emits text with low delay. Google adopted RNN-T for on-device recognition on phones, and it is widely used in real-time products. Training is memory-hungry, because the loss is computed over a grid of every frame against every output position, and several variants exist to reduce this cost or skip frames, such as the token-and-duration transducer used in NVIDIA's Parakeet TDT models.
Attention encoder-decoders#
An attention encoder-decoder (AED) treats recognition like translation. The encoder processes the entire utterance; the decoder generates tokens one at a time, each time attending over all encoder outputs to decide what comes next. Listen, Attend and Spell (Chan and colleagues, 2016) established the approach, and Whisper and NVIDIA's Canary are prominent current examples.
The decoder is a conditional language model, so AEDs produce fluent, well-formatted text and can be trained to handle punctuation, timestamps, language identification and translation in one model through special task tokens. The costs:
- Streaming is awkward. Attention wants the whole input. Chunked and monotonic attention variants exist, but most AEDs are used in batch mode or on short windows.
- Autoregressive decoding is sequential. One decoder step per token makes long transcripts slower than CTC.
- Hallucination. Because output is not tied to specific frames, the decoder can continue generating plausible text that is not in the audio, or repeat a phrase in a loop, especially on silence, music or noise. See hallucinations in speech recognition.
Comparison#
| CTC | Transducer | Attention encoder-decoder | |
|---|---|---|---|
| Output depends on previous tokens | No | Yes, via predictor | Yes, fully |
| Decoding | Parallel, one pass | Frame-synchronous loop | Token-by-token loop |
| Streaming | Yes, with causal encoder | Yes, the usual choice | Difficult |
| Speed | Fastest | Fast | Slowest for long outputs |
| Alignment to audio | Tight, per frame | Tight, per frame | Implicit |
| Typical failure | Misspellings, less fluent text | Deletions under latency pressure | Hallucinated or repeated text |
| Examples | wav2vec 2.0 CTC, Parakeet CTC | Google on-device ASR, Parakeet TDT | Whisper, Canary |
Encoders and hybrids#
The encoder choice is largely shared across families. Recurrent LSTMs gave way to Transformers after 2017, and then to the Conformer (Gulati and colleagues, 2020), which interleaves convolution with self-attention and became the default in many toolkits. Efficiency-focused variants downsample time more aggressively or restructure attention to handle long inputs cheaply. For streaming, the encoder is restricted to past frames plus a small lookahead, usually by masking attention into chunks.
Hybrids combine the strengths of the families. Joint CTC and attention training, described by Watanabe and colleagues in 2017, uses a CTC head on the encoder to enforce monotonic alignment while the attention decoder provides fluency; the CTC score can also veto attention hypotheses that drift from the audio. Another common pattern trains one encoder with both a CTC head and a transducer head, using the fast CTC path for batch jobs and the transducer for streaming.
Choosing an architecture#
- Live voice agents and captions: transducer or chunked CTC, for predictable low latency and output tied to the audio.
- High-volume batch transcription: CTC or transducer for throughput; AED where formatting and multi-task output matter more than cost. See real-time factor.
- Multilingual and translation tasks: AEDs handle task switching through prompting most naturally. See multilingual speech recognition.
- Noisy or silent-heavy audio: frame-aligned models are less prone to hallucination; with an AED, add voice activity detection in front.
Common questions#
Which architecture is most accurate?
None consistently. Leaderboards such as the Open ASR Leaderboard include strong models from all three families. Data and training matter more than the decoder type.
Is Whisper a CTC model?
No. Whisper is an attention encoder-decoder: a Transformer encoder over log-mel features and a Transformer decoder that generates tokens autoregressively.
Why are transducers called RNN-T if they do not use RNNs?
The name comes from Graves's 2012 formulation, which used recurrent networks. The training objective and joiner structure stayed the same when the components were replaced with Transformers, Conformers or stateless predictors.
Can one model do both streaming and batch?
Yes. Models trained with variable attention context can run with limited lookahead for streaming and full context for batch, trading latency for accuracy at inference time.
References#
- Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006). Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. ICML.
- Graves, A. (2012). Sequence Transduction with Recurrent Neural Networks. arXiv:1211.3711.
- Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016). Listen, Attend and Spell. ICASSP.
- Watanabe, S., Hori, T., Kim, S., Hershey, J. R., & Hayashi, T. (2017). Hybrid CTC/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8).
- He, Y. et al. (2019). Streaming end-to-end speech recognition for mobile devices. ICASSP.
- Gulati, A. et al. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. Interspeech.
- Xu, H. et al. (2023). Efficient Sequence Transduction by Jointly Predicting Tokens and Durations. ICML.
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.