A hallucination in speech recognition is text that appears in the transcript without being spoken. It differs from an ordinary mis-recognition, where a real word is heard as a similar-sounding one: a hallucination is invented, often fluent and grammatical, and frequently appears over silence, music or noise. It is mostly a failure of models with autoregressive decoders, and it is dangerous because it looks exactly like correct output.
Key takeaways
- Hallucinations are insertions with no acoustic basis: whole phrases over silence, repeated sentences, or plausible continuations of what came before.
- They are most common in attention encoder-decoder models such as Whisper, whose decoder behaves like a language model. CTC and transducer models are far less prone to them.
- Silence, music, long pauses and the end of an audio chunk are the main triggers.
- Voice activity detection before decoding, and loop detection after it, remove most of them.
- Word error rate counts them as insertions but does not flag them, so a low average WER can hide rare, serious ones.
What a hallucination looks like#
Most recognition errors are acoustically plausible. metformin becomes met for him; fifteen becomes fifty. The words were said, and the model mapped them to the wrong text. A hallucination has no such anchor. Common forms:
- Phantom phrases over non-speech. Ten seconds of hold music transcribed as Thank you for watching! Phrases like this are widely reported for Whisper because they are frequent at the end of video subtitles in web training data.
- Repetition loops. I'll check that for you. I'll check that for you. I'll check that for you. The decoder latches onto its own previous output.
- Plausible continuations. After a speaker trails off, the model completes the sentence with words that fit the context but were never said.
- Language or script switches. A short, noisy clip in Dutch transcribed as an English or even a translated sentence, because the model misidentified the language.
The third kind is the most harmful. A 2024 study by Koenecke and colleagues, Careless Whisper, examined Whisper transcriptions of speech from people with and without aphasia and found that roughly one per cent contained hallucinated phrases, some of them inventing violent content, medical details or relationships. Hallucinations were more frequent for speakers with longer pauses, which ties the failure to non-speech gaps rather than to the words themselves.
Why models hallucinate#
An attention encoder-decoder generates text one token at a time, each conditioned on the audio encoding and on the tokens already produced. The decoder is, in effect, a language model steered by audio. When the audio carries strong evidence, the steering dominates. When it carries little, as in silence or music, the language model prior takes over and produces the most probable text given the training data. For a model trained on web video subtitles, the most probable text over silence at the end of a clip is a sign-off.
Nothing in the architecture forces the output to be aligned with the input. The decoder can emit fifty tokens for one second of audio, or zero for ten. Loops arise when attention stops advancing through the audio and keeps attending to the same frames, so the decoder keeps producing the same text.
CTC and RNN-T models are structurally different. They emit output frame by frame in a monotonic alignment with the audio, and silence frames are mapped to a blank symbol. They can still mis-hear words, and a strong internal language model in a transducer can still pull output towards common phrases, but they rarely generate long, fluent passages from nothing.
Common triggers#
| Trigger | Typical symptom | Main mitigation |
|---|---|---|
| Silence or hold music | Sign-off phrases, credits, "Thank you." | VAD gating |
| Long pauses mid-utterance | Invented continuation of the sentence | VAD, shorter segments |
| Chunk boundaries in long audio | Repeated or duplicated sentences | Overlap handling, loop detection |
| Loud noise, crosstalk | Unrelated fluent text | Denoising, confidence thresholds |
| Very short clips | Wrong language, generic phrases | Fix the language, pad or merge clips |
| Over-strong prompts or biasing | Prompt text copied into output | Weaker biasing, see over-biasing |
The last row matters for context biasing. Prompting an encoder-decoder model with a list of terms can cause it to emit those terms when the audio is ambiguous, or to copy the prompt verbatim. That is a form of over-biasing.
Detection and mitigation#
Before decoding#
The single most effective step is to stop feeding non-speech to the decoder. A voice activity detector such as Silero VAD segments the audio and only speech regions are transcribed. WhisperX, for example, uses VAD-based segmentation and reported fewer hallucinations and repetitions as a result. Segmenting at pauses rather than at fixed intervals also avoids cutting words in half, which is another source of invented text in long-form transcription.
During decoding#
Whisper's reference implementation ships with three heuristics that are worth knowing because many derived systems copy them. A segment is re-decoded at a higher temperature if its gzip compression ratio exceeds 2.4 (a sign of looping text) or its average log-probability falls below −1.0. A segment is dropped as silence if the model's no-speech probability exceeds 0.6 and the log-probability is also low. Conditioning on previous text can be switched off, which reduces loops that propagate from one window to the next at a small cost in consistency.
After decoding#
- Flag segments whose word rate is implausible for their duration: forty words in three seconds is not speech.
- Detect exact or near-exact repeated n-grams within a segment.
- Compare word timestamps with VAD regions. Words timed inside non-speech are suspect.
- Use confidence scores with care. Hallucinated text is often produced with high confidence, because it is exactly what the language model prior expects.
Measuring hallucinations#
WER counts hallucinated words as insertions, so a long invented passage raises the score. But it averages them in with every other error, and a phrase inserted once in a hundred files barely moves a corpus figure. Useful supplementary measures are the insertion rate reported on its own, the rate of words produced on non-speech audio, and the share of files containing any run of consecutive insertions longer than a few words. For regulated uses, human review of a sample of transcripts with long insertion runs is still the most reliable check. See evaluation beyond WER.
Common questions#
Is a hallucination the same as a transcription error?
It is one kind. Most errors are substitutions and deletions of words that were spoken. Hallucinations are insertions of text with no basis in the audio. The distinction matters because the fixes differ: better acoustics help mis-hearings, VAD and decoding controls help hallucinations.
Do all speech models hallucinate?
Any model can insert a stray word. Long, fluent invented passages are characteristic of autoregressive encoder-decoder models. Frame-synchronous CTC and transducer models rarely produce them.
Does a lower temperature stop hallucinations?
Not reliably. Greedy decoding at temperature zero is what produces many repetition loops; Whisper's fallback actually raises the temperature to escape them. Removing non-speech from the input is more effective than tuning sampling.
Why are hallucinations worse in medical or legal settings?
Because they are fluent and plausible, a reviewer may not notice them. An invented dosage or statement in a clinical note or court record is harder to catch than an obviously garbled word.
References#
- Koenecke, A., Choi, A. S. G., Mei, K. X., Schellmann, H., & Sloane, M. (2024). Careless Whisper: Speech-to-Text Hallucination Harms. ACM FAccT.
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.
- Bain, M., Huh, J., Han, T., & Zisserman, A. (2023). WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. Interspeech.
- OpenAI Whisper reference implementation. GitHub.
- Silero VAD: pre-trained enterprise-grade voice activity detector. GitHub.
- Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006). Connectionist Temporal Classification: labelling unsegmented sequence data with recurrent neural networks. ICML.