Word error rate is the standard measure of speech recognition accuracy, and a poor summary of whether a transcript is fit for use. It weights every word equally, ignores formatting and speakers, says nothing about speed, and averages away the failures that matter most. A useful evaluation adds a small set of targeted measures: errors on the words your application depends on, whether meaning survives, who said what, how quickly results arrive, and how accuracy varies across speakers and conditions.
Key takeaways
- Two transcripts with identical WER can differ completely in usefulness. Measure the errors that affect your task, not only the total.
- Entity and keyword accuracy, semantic error measures and formatting accuracy each capture something WER misses.
- For multi-speaker audio, measure attribution as well as words: DER for diarization, cpWER for both together.
- Latency and hallucination rate belong in the same report as accuracy.
- Always break results down by condition. An average can hide a subgroup where the system fails.
Same WER, different outcome#
WER counts substitutions, deletions and insertions and divides by the number of reference words. It does not ask which words were wrong.
Aggregate WER over thousands of utterances hides the same problem at scale. A system can reach a low corpus WER by getting common words right and still miss most drug names, product codes or surnames, because those are rare and contribute little to the total.
A toolkit of complementary metrics#
| Metric | What it measures | Use it when |
|---|---|---|
| Entity / keyword error rate | Recall and precision on a list of important terms or entity spans | Names, drugs, products or codes drive the workflow |
| Semantic WER and embedding distances | Whether errors change meaning | A language model or human acts on the transcript's meaning |
| CER | Character-level edit distance | Languages without spaces, heavy compounding, identifiers |
| MER, WIL | Bounded alternatives to WER | Comparing systems with many insertions |
| Formatted WER, punctuation F1 | Casing, punctuation, number formatting | Transcripts are read by people, or parsed |
| DER, cpWER | Speaker attribution | Meetings, calls, interviews |
| Latency (p50, p95) | Time from speech to usable text | Any real-time use |
| Insertion rate on non-speech | Hallucinated text | Audio contains silence, music or noise |
Important words#
The most direct fix for equal weighting is to measure the words that carry the value. Build a list of domain terms or annotate entity spans in your references, then report how many were transcribed exactly. Report precision as well as recall: context biasing can raise recall on target terms while inserting them where they were not said, a failure known as over-biasing. For identifiers such as order numbers and licence plates, exact-match accuracy of the whole string is usually the only measure that matters; see structured identifiers.
Meaning#
Semantic measures try to count only errors that change meaning. Approaches range from rules that ignore harmless variants (going to vs gonna) to embedding distances between reference and hypothesis sentences, as in the semantic distance metric proposed by Kim and colleagues in 2021, to judgements by a language model. They correlate better with downstream task success than raw WER, but they depend on the model or rules used to judge, so fix the method and report it.
Formatting#
Standard WER is computed after normalisation strips case, punctuation and number formatting. That is correct for comparing recognisers and blind to readability. When transcripts are read by people or parsed by software, also score the unnormalised output against a written-form reference. See inverse text normalisation.
Speakers#
For multi-speaker audio, the question is not only what was said but who said it. Diarization error rate measures the fraction of time attributed to the wrong speaker or missed. cpWER, used in the CHiME-6 challenge, computes WER per speaker after finding the best mapping between hypothesis and reference speakers, so a correctly recognised sentence attributed to the wrong person still counts as errors. See speaker diarization.
Speed and reliability#
Accuracy without timing is incomplete for any live application. Report latency as a distribution, measured at the client, on the same run as accuracy, because many systems trade one for the other. For batch workloads, report real-time factor or wall-clock turnaround. Reliability also deserves its own numbers: the rate of hallucinated insertions on non-speech audio, the share of files with long insertion runs, and the failure rate of the service itself.
Breaking results down#
A single average hides variation between speakers and conditions. In 2020, Koenecke and colleagues tested five commercial speech recognition systems on interviews with Black and white American speakers and found substantially higher WER for Black speakers across all five, a disparity that no overall average would show. The same principle applies to any subgroup that matters to you: accents, age groups, devices, channels, noise levels, languages. Report WER and entity accuracy per stratum, with the number of samples in each, and look at the worst stratum as well as the mean. See accents and regional speech.
Task-based evaluation#
The most reliable measure is often the downstream task itself. If transcripts feed a summariser, score the summaries. If they fill fields in a CRM, measure field accuracy. If a voice agent acts on them, measure task completion. Task metrics are expensive and specific, which is why they complement rather than replace WER: WER is cheap, repeatable and comparable across systems; task metrics tell you whether the differences matter. Building your own test set from real production audio makes both kinds meaningful.
How Reson8 approaches this#
Reson8 reports targeted metrics next to aggregate WER. The Turns endpoint is measured on semantic WER (1.10% pooled, 1.39% mean on the Pipecat STT benchmark) together with a median 326 ms from end of speech to final transcript, because a voice agent needs both. Patterns is measured on identifier WER, where order numbers fell from 8.0% to 1.5%. Aggregate WER on public sets, with the method and exclusions stated, is in the benchmarks post.
Common questions#
Should I stop using WER?
No. It is cheap, standard and comparable across systems when computed with the same normaliser on the same data. Use it as the baseline and add the measures your use case needs.
Which extra metric should I add first?
For most business applications, accuracy on a list of important terms or entities. It is simple to compute and usually reveals the largest differences between systems.
Can an LLM judge transcript quality?
It can estimate whether meaning was preserved and is useful for triage at scale, but its judgements vary with the prompt and model. Fix both, validate against human ratings on a sample, and report the method.
How many test files do I need per subgroup?
Enough that the error counts are not dominated by a few files. Tens of minutes of audio per stratum is a practical minimum for spotting large gaps; small differences need more.
References#
- Morris, A. C., Maier, V., & Green, P. (2004). From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition. Interspeech.
- Koenecke, A. et al. (2020). Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14).
- Kim, S. et al. (2021). Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding. Interspeech.
- Watanabe, S. et al. (2020). CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings. arXiv:2004.09249.
- NIST Scoring Toolkit (SCTK). GitHub.
- Bredin, H. et al. (2020). pyannote.audio: neural building blocks for speaker diarization. ICASSP.