ASR benchmarks are fixed, public test sets with reference transcripts, used to compare speech recognition systems on equal terms. The most cited today are the English sets collected in the Hugging Face Open ASR Leaderboard, which cover audiobooks, talks, parliament speeches, earnings calls and meetings, and FLEURS for multilingual evaluation. They are the best shared yardstick the field has, but each tests a particular kind of speech, and none is your audio.
Key takeaways
- Each benchmark samples one speaking style. Read audiobooks and spontaneous meetings differ by several points of WER for the same model.
- Leaderboards average across sets, so the average depends on which sets are included and how they are weighted.
- Scores are only comparable under the same normaliser, the same split and the same scoring script. Leaderboards fix all three, which is their main value.
- Public test sets can leak into training data. A score on a contaminated set says little about generalisation.
- Use benchmarks to shortlist. Use your own test set to decide.
What a benchmark contains#
A speech benchmark has three parts: audio, a reference transcript for every clip, and a scoring procedure. The scoring procedure is the part most often ignored. It specifies the text normaliser, whether scores are pooled across the corpus or averaged per clip, and how long files are segmented. Change any of these and the number moves, sometimes by more than the gap between two competing models.
Most academic corpora also define training, development and test splits. Only the test split is used for reported results. The development split is for tuning. Mixing them up, or training on data that overlaps the test split, invalidates the result; see benchmark contamination.
The main English test sets#
The Open ASR Leaderboard grew out of the ESB benchmark proposed by Gandhi, von Platen and Rush, which argued that systems should be tested across several domains at once rather than tuned to one. Its short-form English track uses the following sets.
| Dataset | Source audio | Speaking style | What it stresses |
|---|---|---|---|
| LibriSpeech (clean, other) | LibriVox public-domain audiobooks | Read | Clean close-mic speech; "other" has harder speakers |
| TED-LIUM | TED talks | Prepared, spoken | Lecture-style delivery, varied accents |
| VoxPopuli | European Parliament sessions | Prepared and semi-spontaneous | Non-native English, political vocabulary |
| GigaSpeech | Audiobooks, podcasts, YouTube | Mixed | Breadth of domain and recording conditions |
| SPGISpeech | Corporate earnings calls | Prepared and Q&A | Financial terms, numbers, formatted text |
| Earnings-22 | Earnings calls from companies worldwide | Prepared and Q&A | Accented English, company names |
| AMI | Recorded business meetings | Spontaneous | Overlap, disfluency, meeting-room acoustics |
The spread matters. Strong models reach low single-digit WER on LibriSpeech test-clean, while the same models are often several times worse on AMI. A model tuned to shine on read audiobooks can rank poorly on meetings, which is why the leaderboard averages across sets. It is also why the average depends on the set list: adding or removing LibriSpeech, the easiest set, shifts everyone's mean.
The leaderboard reports RTFx next to WER, measured on a stated GPU. Accuracy and speed are traded against each other in model design, so a ranking on WER alone favours the largest, slowest models. Later versions of the leaderboard added multilingual and long-form tracks, which test different failure modes: language coverage and drift over hours of audio.
Multilingual benchmarks#
FLEURS, released by Google in 2022, has around 12 hours of read speech in each of 102 languages. Every language reads translations of the same sentences, taken from the FLoRes-101 machine translation benchmark, so results are comparable across languages in a way most corpora are not. The trade-off is that the speech is read, fairly clean and short. It says little about conversation or telephony.
Other sets fill gaps. Multilingual LibriSpeech provides read audiobooks in eight European languages. Mozilla Common Voice is crowd-sourced read speech in many languages, with wide speaker and microphone variety but short, often formulaic sentences. VoxPopuli includes transcribed parliament speech in several EU languages. For European deployment, a model's results on FLEURS in the languages you need are a reasonable first filter.
How to read a leaderboard#
- Check the set list. An average over five sets is not comparable with an average over seven. Ask which sets were excluded and why.
- Look at the per-set columns. The set closest to your audio matters more than the mean. For call-centre work, AMI and Earnings-22 are more informative than LibriSpeech.
- Consider what the model was trained on. Many public training sets have matching test splits. A model trained on the LibriSpeech training split is being tested in-domain on LibriSpeech test.
- Treat small gaps as ties. Test sets have finite size. A few tenths of a point between two models may fall inside the noise, especially on the smaller sets.
- Match the mode. Most leaderboard numbers are from offline, full-context decoding. Streaming accuracy is usually lower and is rarely reported on the same table.
How Reson8 approaches this#
As of April 2026, Resonant-1 had the lowest average WER on the Open ASR Leaderboard short-form English sets it was evaluated on: 6.51% average across AMI (9.39%), Earnings22 (9.02%), GigaSpeech (8.76%), SPGISpeech (3.05%) and TED-LIUM (2.32%), at RTFx 1187. LibriSpeech and VoxPopuli were left out because they were used in training; including them would have inflated the score, and training on them was also found to hurt generalisation. On long-form English, its 8.58% average placed third, behind ElevenLabs Scribe v2 (7.32%) and AssemblyAI Universal 3 Pro (8.34%).
On FLEURS, Resonant-1 averaged 4.22% across English, German, French, Dutch, Spanish and Polish, leading in French, Dutch, Spanish and Polish; Swedish, at 7.38%, is reported separately. Full tables are in the benchmarks post.
Common questions#
Which benchmark is closest to real-world audio?
For English conversation, AMI and Earnings-22 are the closest public options. None covers 8 kHz telephony well. For phone audio you need your own test set; see building an ASR test set.
Why do vendor numbers differ from the leaderboard?
Different normalisers, different test subsets, different decoding settings, or results from an older model version. Vendors may also report on internal sets that cannot be checked. Prefer numbers anyone can reproduce.
Is a lower average WER always better?
Only for audio like the benchmark average. A model that is best on average can still be worse on the one set that resembles your traffic. Read the columns.
Do benchmarks measure speaker diarization or punctuation?
Not the WER leaderboards. They normalise away punctuation and casing, and they do not score who spoke when. Diarization has its own benchmarks and metrics.
References#
- Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). LibriSpeech: an ASR corpus based on public domain audio books. ICASSP.
- Gandhi, S., von Platen, P., & Rush, A. M. (2022). ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition. arXiv:2210.13352.
- Conneau, A. et al. (2023). FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. IEEE SLT 2022.
- Carletta, J. et al. (2005). The AMI Meeting Corpus: A Pre-announcement. MLMI.
- O'Neill, P. K. et al. (2021). SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition. Interspeech.
- Chen, G. et al. (2021). GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio. Interspeech.
- Wang, C. et al. (2021). VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation. ACL.
- Srivastav, V. et al. Open ASR Leaderboard. Hugging Face.