Most speech recognition benchmarks use fairly clean audio. Production audio is not clean: it comes from a phone in a car, a laptop across a meeting room, a headset in an open-plan contact centre. Noise, reverberation, distance and overlapping talkers are the main reasons a model that scores well on a leaderboard does worse on real recordings. The effective remedies are, in order, better capture, models trained on realistic conditions, and careful front-end processing; aggressive noise suppression can make things worse.
Key takeaways
- "Noisy" covers different problems: additive noise, reverberation, distance from the microphone, and other people talking. Each degrades recognition differently.
- Overlapping speech and far-field reverberation are harder for modern models than steady background noise.
- Microphone placement is the largest single factor. Halving the distance to the talker helps more than most software.
- Noise suppression tuned for human listeners can remove cues the recogniser needs. Test with and without it.
- Evaluate on your own noisy audio. Clean-speech benchmark scores do not transfer.
What "noisy" means#
A microphone records the sum of everything reaching it. For a speech recogniser, only one part of that is wanted.
The standard measure is signal-to-noise ratio (SNR) in decibels. A close-talk headset in a quiet office might sit above 30 dB; a phone on speaker in a moving car can drop to single figures. But SNR alone does not predict accuracy, because the kind of interference matters as much as its level.
Types of interference#
| Type | Examples | Effect on recognition | What helps |
|---|---|---|---|
| Stationary noise | Fans, engine hum, air conditioning | Masks quiet phonemes; modern models are fairly tolerant | Training augmentation, closer microphone |
| Non-stationary noise | Traffic, keyboards, doors, dishes | Bursts mask whole words; can trigger insertions | VAD, augmentation with real noise |
| Babble | Cafés, open-plan offices, contact centres | Sounds like speech, so the model transcribes fragments of it | Directional microphones, headsets |
| Reverberation | Meeting rooms, halls, bathrooms | Smears phonemes into each other over hundreds of milliseconds | Near-field capture, dereverberation, beamforming |
| Overlapping speech | Meetings, interruptions, crosstalk | Deletions and merged words; hardest case | Separate channels, separation models |
| Channel distortion | Low-bitrate codecs, clipping, packet loss | Removes information outright | Better capture and encoding |
Far-field and reverberation#
In a typical room, the sound reaching a microphone a few metres away is dominated by reflections rather than the direct path. Each phoneme is followed by decaying copies of itself that overlap the next one. Human listeners compensate using two ears and visual cues; a single microphone cannot. The AMI meeting corpus records the same meetings with close-talk headsets and with distant microphone arrays, and word error rates on the distant channels are consistently and substantially higher for every system evaluated on it.
Overlap#
When two people speak at once, a single-output recogniser has to choose, and it often produces a blend of both or drops one. In natural conversation overlap is common at turn boundaries and in backchannels such as "yeah" and "mm-hm". The CHiME-5 and CHiME-6 challenges, recorded at real dinner parties in people's homes with distant microphones, were designed around exactly this and remain difficult for all systems. See speaker diarization for the related problem of attributing speech to speakers.
Making models robust#
Modern models gain most of their robustness from training data. Two approaches dominate:
- Data augmentation. Clean training audio is mixed with recorded noise at varying SNRs and convolved with room impulse responses to simulate reverberation. SpecAugment, which masks bands of frequency and spans of time in the spectrogram, is a cheap and widely used complement.
- Diverse real data. Large-scale weakly supervised training on varied web audio, as in Whisper, exposes the model to many recording conditions. The Whisper paper reported notably better robustness to added noise than models trained only on LibriSpeech, even where their clean-speech scores were similar.
The remaining weak points are the ones least represented in any training set: heavy overlap, distant microphones in reverberant rooms, and noise that resembles speech.
Front-end processing#
Processing before recognition can help or hurt. The general rule is that anything which increases the share of direct speech helps, and anything which alters speech to sound cleaner to a human may not.
- Voice activity detection is almost always worth it. Removing non-speech stops the recogniser from transcribing noise, a common source of hallucinations. See VAD.
- Beamforming with a microphone array reinforces the talker's direction. It is the standard approach in meeting-room devices and cars, and one of the few front-end steps that reliably improves recognition in far-field conditions.
- Single-channel noise suppression, of the kind built into conferencing and phone software, is tuned for human comfort. It can introduce artefacts, clip word onsets, or remove consonant energy the model relies on. Strong end-to-end models trained on noisy data sometimes do better on the unprocessed signal.
- Automatic gain control can amplify background noise in pauses. Moderate settings are fine; aggressive ones are not.
Capture advice#
- Put the microphone close to the talker. A headset or lapel microphone beats any room microphone.
- Record each participant on a separate channel when possible, and transcribe the channels separately.
- Use directional microphones in noisy environments, pointed at the talker and away from the noise.
- Avoid hard, bare rooms for recorded interviews; soft furnishings reduce reverberation.
- Keep levels well below clipping. Distortion is harder to recover from than low level.
Evaluating on noisy audio#
Public benchmarks differ widely in difficulty. Read audiobooks are clean; AMI meetings and CHiME dinner parties are not. A test set of your own should sample the actual conditions: the same devices, rooms and background. Stratify results by condition, reporting WER separately for quiet and noisy calls or near and far microphones, because an average across both hides where a system fails. Track insertions separately too, since noise tends to produce them.
How Reson8 approaches this#
Reson8 reports Resonant-1 results on the Open ASR Leaderboard sets that include real-world recording conditions, among them AMI meeting speech (9.39% WER) and Earnings22 conference calls (9.02%), alongside cleaner sets such as TED-LIUM (2.32%). The gap between those figures is a fair indication of how much harder meeting and call audio is than prepared talks. Full tables are in the benchmarks post.
Common questions#
Should I denoise audio before transcription?
Not by default. Test it on your data. Light processing and beamforming often help; heavy single-channel suppression can remove information and increase errors with modern models.
What signal-to-noise ratio is good enough?
There is no single threshold, because noise type matters as much as level. As a rough guide, clean headset recordings are well above 20 dB, and accuracy degrades noticeably once speech and noise are close in level.
Why are meeting recordings so hard?
They combine distance, reverberation, overlapping speakers and informal speech. Each alone is manageable; together they compound.
Does background music cause problems?
Music with vocals is similar to babble and can be partially transcribed. Instrumental music mostly masks speech, and over silence it can trigger hallucinations in some models.
References#
- Carletta, J. et al. (2005). The AMI Meeting Corpus: A Pre-announcement. Machine Learning for Multimodal Interaction (MLMI).
- Barker, J., Watanabe, S., Vincent, E., & Trmal, J. (2018). The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines. Interspeech.
- Park, D. S. et al. (2019). SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. Interspeech.
- Ko, T., Peddinti, V., Povey, D., Seltzer, M. L., & Khudanpur, S. (2017). A study on data augmentation of reverberant speech for robust speech recognition. ICASSP.
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.
- Lombard, É. (1911). Le signe de l'élévation de la voix. Annales des Maladies de l'Oreille et du Larynx, 37.