Benchmark contamination happens when material from a test set, or from the same source as a test set, ends up in a model's training data. The model then scores better on that benchmark than it would on new audio of the same kind, and the benchmark stops measuring generalisation. In speech recognition the risk is high because the most popular test sets are built from public audiobooks, parliament recordings, talks and earnings calls that also circulate widely on the web.
Key takeaways
- Contamination is not only copying the test split. Training on a corpus's own training split, or on web data drawn from the same source, also inflates scores on that corpus.
- The effect is a gap between benchmark WER and real-world WER. A model can look excellent on LibriSpeech and ordinary on meetings.
- Text alone can contaminate. A decoder that has seen the reference sentences can predict them from weak acoustic evidence.
- Honest reporting states which training data overlaps which test sets, and excludes or flags those results.
- The only reliable defence for a buyer is a private test set that no vendor has seen.
Forms of contamination#
Direct inclusion#
The test audio and its transcripts are in the training set. This is rare in careful work and usually accidental, for example when a large dataset is assembled by merging public corpora without removing their test splits.
In-domain training#
The model is trained on a corpus's training split and evaluated on its test split. This is standard academic practice and is not wrong in itself. But it means the model has learned the corpus's recording conditions, its vocabulary and, importantly, its transcription conventions. The resulting score describes performance on that corpus, not on speech in general. When a leaderboard mixes models trained in-domain with models that were not, it compares different things.
Shared sources#
Many benchmarks are cut from material that is public by design. LibriSpeech comes from LibriVox audiobooks. VoxPopuli comes from European Parliament recordings. TED-LIUM comes from TED talks, which are widely subtitled online. Earnings calls are published by the companies holding them. A model trained on large amounts of web audio with captions may well have heard the same recordings, or other readings of the same book, without anyone intending it.
Text leakage#
Modern end-to-end models contain an implicit language model, and some systems add an external one. If the reference sentences were in the text used to train it, the model can complete a sentence from a few clear words. FLEURS, for instance, is built on sentences from the FLoRes machine translation benchmark, which were drawn from English Wikimedia projects. A decoder trained on web text may have seen those sentences, in English and in translation.
Why it inflates scores#
Contamination rewards memory and familiarity rather than recognition. The model has heard the speakers, knows the phrasing, or has learned the exact way the reference transcribers wrote numbers and hesitations. Each of these lowers WER on the benchmark without improving the model on new audio.
The Whisper paper made the related point about in-domain training forcefully. It observed that models trained on LibriSpeech achieve very low WER on LibriSpeech yet make far more errors on other datasets, while a model trained on broad data without the LibriSpeech training split did much better across distributions for the same LibriSpeech score. Likhomanenko and colleagues reached a similar conclusion: models trained on a single public corpus generalise poorly, and training on a mix of sources gives more reliable results on unseen sets.
Computer vision shows how far a fixed test set can drift from reality even without contamination. Recht and colleagues built fresh test sets for CIFAR-10 and ImageNet following the original collection procedures and found that accuracy fell for every model they tested, by roughly 11 to 14 points on ImageNet, although the ranking of models stayed largely the same. They attributed the drop mainly to small differences in the new data rather than to repeated test-set reuse. The lesson for speech is that a benchmark score is an optimistic estimate even when the data is clean, and contamination makes it more optimistic still.
Detecting it#
A model developer can check for most forms of contamination. An outside evaluator usually cannot see the training data and must rely on indirect signals.
| Check | Who can run it | What it catches |
|---|---|---|
| Hash or fingerprint test audio against training audio | Developer | Direct inclusion, duplicate recordings |
| Match speaker, book or session IDs against training manifests | Developer | Shared sources, same speakers |
| Long n-gram overlap between test references and training text | Developer | Text leakage |
| Compare public-set WER with WER on a fresh set of the same style | Anyone | Any inflation, whatever its cause |
| Look for verbatim completion of long reference sentences from noisy audio | Anyone | Memorised text |
The fresh-set comparison is the most useful for buyers. Record or collect audio in the same style as a public benchmark, transcribe it carefully, and see whether each system's gap between the two is similar. A system whose public score is much better than its fresh score, relative to others, has probably seen the public set in some form.
Reporting honestly#
Good practice for anyone publishing ASR results:
- List the training corpora that have matching test splits, and mark results on those sets as in-domain.
- Report averages both with and without the affected sets, or exclude them and say so.
- Publish the normaliser and scoring script, so the absence of contamination is the only open question.
- Add at least one evaluation on data collected after the training cut-off.
For readers, the question to ask of any leaderboard number is simple: could this model have seen this data, or data from the same source? If the answer is yes or unknown, treat the number as an upper bound on quality.
How Reson8 approaches this#
Resonant-1 was trained with LibriSpeech and VoxPopuli data, so Reson8 excludes both from its reported Open ASR Leaderboard average. The published 6.51% covers the other five short-form English sets: AMI, Earnings22, GigaSpeech, SPGISpeech and TED-LIUM. Reson8 also found that training on those two corpora hurt generalisation, which matches the pattern described above. The method is stated in the benchmarks post.
Common questions#
Is training on a benchmark's training split cheating?
No. It is how most corpora are meant to be used. It becomes misleading when the resulting in-domain score is compared with zero-shot scores from other models without saying so.
Can contamination make a model worse?
On the contaminated benchmark, no; it helps there. On other audio it can, if the contaminating data crowded out more varied material or taught narrow conventions.
How do I know whether a vendor's model has seen a benchmark?
Ask which public corpora were used in training and whether results on those corpora are excluded. Then test on a private set. A vendor that cannot answer the first question should be judged on the second alone.
Does contamination affect FLEURS?
It can. Its sentences come from public text, and its audio has been public since 2022. Treat FLEURS results as indicative, and confirm on your own recordings in each language.
References#
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.
- Likhomanenko, T. et al. (2021). Rethinking Evaluation in ASR: Are Our Models Robust Enough? Interspeech.
- Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019). Do ImageNet Classifiers Generalize to ImageNet? ICML.
- Magar, I., & Schwartz, R. (2022). Data Contamination: From Memorization to Exploitation. ACL.
- Sainz, O. et al. (2023). NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Findings of EMNLP.
- Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). LibriSpeech: an ASR corpus based on public domain audio books. ICASSP.