Semantic word error rate is a family of metrics that score a transcript only on the errors that change what it means. It starts from the same alignment as word error rate, then discards edits a careful reader would treat as equivalent, such as a dropped "um" or "gonna" for "going to". It is most useful where transcripts feed a machine, as in voice agents, and least useful where exact wording is the product, as in legal records.
Key takeaways
- Semantic WER keeps the WER formula but changes what counts as an error: only edits that alter meaning are scored.
- There is no single standard. Implementations use rule-based equivalence lists, embedding similarity, or a language model acting as judge, and they produce different numbers.
- It is lower than plain WER on the same audio, often by a wide margin on conversational speech, because much of conversational WER is fillers, contractions and hesitations.
- It suits voice agents and analytics, where a downstream model reads the text. It suits verbatim transcription poorly.
- A semantic WER figure is only comparable with another produced by the same judge and the same prompt or rules.
Why plain WER over-counts#
WER treats every word edit as equally wrong. On read speech that is a reasonable simplification. On conversation it is not, because spontaneous speech is full of material that carries no content: fillers, false starts, repetitions, and informal forms that a reference transcriber may or may not have written down.
Consider a caller booking an appointment. The reference says um I I want to move it to Thursday and the system outputs I want to move it to Thursday. That is two deletions over nine reference words, a WER of 22%, and a transcript that loses nothing. Now take I can't do Thursday heard as I can do Thursday. One substitution over four words, 25%, and the meaning is reversed. The two scores are nearly identical. Their consequences are not.
Semantic WER tries to separate these cases. It is a response to a problem described in evaluation beyond WER: aggregate word counts hide which errors matter.
How it is computed#
Most implementations share a pipeline. Both texts are normalised, aligned word by word as for WER, and then each edit, or each utterance, is judged for whether meaning survived.
Here Sm, Dm and Im are the substitutions, deletions and insertions judged to change meaning, and N is still the number of reference words. Keeping the reference length as the denominator makes the number directly comparable in scale with WER on the same set.
Three ways to judge an edit#
- Rules and equivalence lists. Extend the normaliser with lists of fillers to ignore and forms to merge: "gonna" and "going to", "OK" and "okay", "10" and "ten". Cheap, deterministic and auditable, but every list is incomplete and language-specific.
- Embedding similarity. Encode reference and hypothesis with a sentence encoder and measure the distance. Kim and colleagues proposed a semantic distance metric along these lines and showed it tracked downstream language understanding better than WER. Similarity scores are continuous, so they must be thresholded or reported separately rather than slotted into an error count.
- A language model as judge. Give an LLM both texts and ask it to identify the errors that change meaning. This handles negation, numbers and paraphrase well, but it adds variance, cost and a dependency on a specific model and prompt.
What counts as meaning-changing#
The line between "equivalent" and "different" is a policy decision, and it should be written down before scoring. A typical policy for a voice agent looks like this:
| Reference | Hypothesis | WER edits | Semantic verdict |
|---|---|---|---|
| uh I'd like to cancel | I'd like to cancel | 1 | Equivalent |
| we're gonna be late | we are going to be late | 4 | Equivalent |
| I can't make Friday | I can make Friday | 1 | Changed: negation lost |
| order fifteen units | order fifty units | 1 | Changed: quantity |
| book with Dr Visser | book with Dr Fisher | 1 | Changed: wrong person |
| yeah that's fine | yes that's fine | 1 | Equivalent |
The pattern is that negations, numbers, names and identifiers always count, while disfluencies and register differences usually do not. Names are the hard case. A misspelt surname may be harmless in a meeting summary and fatal in a prescription, so the policy has to follow the use case.
When to use it, and when not to#
Semantic WER is the better headline metric when a transcript is consumed by software rather than read verbatim. In a voice agent, the LLM that reads the transcript does not care whether the caller said "um". It cares whether "not" survived. The same applies to intent routing, search and summarisation.
It is the wrong metric when the exact words are the deliverable. Legal transcripts, subtitles and clinical dictation are judged on fidelity, and a hesitation can be part of the record. There, plain WER with a documented normaliser, plus targeted entity accuracy, remains the right pair.
It is also a poor metric for comparing systems that differ mainly in style. A model trained to output clean, disfluency-free text will look much better on semantic WER than on WER against a verbatim reference. Both numbers are informative; neither is complete on its own.
Pooled versus mean#
As with WER, a corpus figure can be pooled (sum all meaning-changing errors, divide by all reference words) or averaged per utterance. Voice agent test sets contain many short turns, where a single error in a three-word reply is 33%. The mean is therefore usually higher than the pooled figure, and it better reflects how often a user hits a broken turn. Report both, and say which is which.
How Reson8 approaches this#
Reson8 reports semantic WER for its Turns endpoint, the streaming mode built for voice agents. On the open Pipecat STT benchmark (1,000 samples from smart-turn-data-v3.1), it measured 1.10% pooled semantic WER and 1.39% mean semantic WER, with a median 326 ms from end of speech to final transcript. Reporting the two aggregations side by side follows the reasoning above: the gap between them shows how errors are spread across short turns.
Common questions#
Is semantic WER always lower than WER?
On the same data, yes, provided the judge only removes edits and never adds them. The size of the gap depends on how disfluent the speech is and how verbatim the references are. On read speech the two are close; on conversation the gap can be several points.
Can I compare semantic WER across vendors' published numbers?
Only if they used the same benchmark, the same judge and the same prompt or rules. Otherwise the difference may come from the judge, not the model. Run the comparison yourself on your own audio.
Does semantic WER replace entity accuracy?
No. A good judge will flag a wrong name, but it reports it as one error among many. If names, codes or drug terms decide whether a transcript is usable, measure them separately with entity or keyword metrics.
Why not just use BERTScore?
BERTScore and similar embedding metrics are useful, but they produce a similarity score rather than an error count, and they can rate a sentence with a flipped negation as highly similar to the original. They work best as a secondary signal.
References#
- Kim, S. et al. (2021). Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding. Interspeech.
- Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. ICLR.
- Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP.
- Morris, A. C., Maier, V., & Green, P. (2004). From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition. Interspeech.
- Szymański, P. et al. (2020). WER we are and WER we think we are. Findings of EMNLP.