Entity and keyword accuracy measures how well a speech recognition system transcribes the specific words a use case depends on: names, drug terms, product codes, amounts and dates. It is reported as recall and precision on a list of target terms, as a word error rate restricted to those terms, or as an exact-match rate for identifiers. It exists because overall WER weights every word equally, and in most applications a small set of words carries most of the value.
Key takeaways
- Most words in a transcript are common function words that every modern system gets right. The errors that matter cluster on rare terms.
- Measure both recall (did the spoken term appear?) and precision (did the system insert terms nobody said?). Biasing tends to raise the first and can lower the second.
- Report WER on target terms and WER on everything else separately, so an improvement on one does not hide a regression on the other.
- For identifiers, score the whole code as right or wrong. One wrong character makes an order number useless.
- The target list must be fixed before testing and drawn from real usage, not from the terms the system already handles.
Why overall WER is not enough#
Take a 40-word clinical note containing two drug names. A system that gets every ordinary word right but mishears both drugs scores around 5% WER, which reads as a good result. For the clinician, the transcript is unusable. Another system might miss four filler words and get both drugs right, scoring 10% WER with a transcript that is fit for purpose.
This is not a corner case. Rare terms are, by definition, underrepresented in training data, so they are where a general model is weakest. They are also the words users search for, act on and are held liable for. A drug name heard as a pixie ban instead of apixaban costs one substitution and two insertions in WER, the same as three dropped "the"s.
Research on earnings calls made the pattern visible at scale. The Earnings-21 benchmark tagged named entities in its references and reported error rates by entity class. For the systems it tested, entities such as people and organisations were harder than the surrounding text.
The main metrics#
Recall and precision on a term list#
Start with a list of target terms: product names, staff names, medications, whatever the application cannot get wrong. For each utterance, count how many times each term occurs in the reference and in the hypothesis. The overlap is the number of hits.
Recall is the number most people want: when someone said the term, did it appear? Precision matters as soon as you add custom vocabulary or context biasing, because a biased system can start writing target terms where similar-sounding ordinary words were spoken. That failure is covered in over-biasing. F1, the harmonic mean of the two, is a convenient single figure but hides which way the system is failing.
WER split by word class#
Research on contextual biasing often splits WER in two: WER over words in the biasing list, and WER over all other words. Le and colleagues at Facebook reported this pair as biased and unbiased WER. The split makes trade-offs visible. A biasing method that cuts biased-word WER sharply while raising unbiased WER is moving errors around, not removing them.
Exact match for identifiers#
Order numbers, policy numbers, licence plates and IBANs are right or wrong as a whole. INV7625X transcribed as INB7625X has one wrong character, a character error rate of 12.5%, and zero value to the system trying to look it up. Report the share of identifiers matched exactly, after a normalisation step that removes spaces and fixes case. See structured identifiers and numbers and alphanumerics.
Computing keyword scores#
A per-utterance bag-of-words match is enough for single-word terms. Multi-word terms need matching on n-grams, and inflected languages need a decision about whether "Bakkers" counts as a hit for "Bakker".
from collections import Counter
def keyword_scores(pairs, keywords):
keywords = {k.lower() for k in keywords}
hits = in_ref = in_hyp = 0
for ref, hyp in pairs:
ref_counts = Counter(w for w in ref.lower().split() if w in keywords)
hyp_counts = Counter(w for w in hyp.lower().split() if w in keywords)
hits += sum((ref_counts & hyp_counts).values())
in_ref += sum(ref_counts.values())
in_hyp += sum(hyp_counts.values())
return hits / in_ref, hits / in_hyp if in_hyp else 1.0
pairs = [
("start apixaban five milligrams", "start a pixie ban five milligrams"),
("continue apixaban and bisoprolol", "continue apixaban and bisoprolol"),
("stop the ramipril", "stop the apixaban"),
]
recall, precision = keyword_scores(pairs, ["apixaban", "bisoprolol", "ramipril"])
print(f"recall {recall:.0%} precision {precision:.0%}") # recall 50% precision 67%
The third utterance is the dangerous one: a real drug replaced by a different real drug. It counts as both a miss and a false alarm, and it is exactly the error a word-level WER would score the same as a missed article.
Building the term list#
The metric is only as good as the list. Three rules keep it honest:
- Draw it from usage, not from the system. Take terms from product catalogues, formularies, CRM records and past transcripts corrected by people. A list built from what the system already recognises measures nothing.
- Fix it before testing. Adding terms after seeing results turns evaluation into tuning.
- Separate the biasing list from the test list. If you are evaluating context biasing, include some target terms in the test set that were not in the biasing list. That shows whether the system generalises or only recognises what it was told.
Weighting helps when terms differ in importance. A misheard medication or account number can be weighted above a misheard street name. Keep any weighting simple and published with the result, or the score becomes impossible to interpret.
| Term type | Typical metric | Normalisation to agree |
|---|---|---|
| Drug and clinical terms | Recall, precision | Brand vs generic, spelling variants |
| Person and company names | Recall, precision | Case, diacritics, inflected forms |
| Order and policy numbers | Exact match | Spaces, case, spoken vs written digits |
| Amounts and dates | Exact match on value | "€20" vs "twenty euros" |
How Reson8 approaches this#
Reson8 reports targeted metrics for its targeted features. Patterns, which lets a caller supply an expected identifier format such as INV[0-9]{3}, is measured on identifier WER: order numbers improved from 8.0% to 1.5% and licence plates from 6.6% to 4.7%. Domain adaptation takes domain text through the API, supports lists of 50,000+ medical terms without added latency, and is best evaluated with the recall and precision measures described here on your own term list.
Common questions#
What is a good keyword recall?
There is no general threshold. It depends on the cost of a miss and whether a person reviews the output. Compare systems on the same list and the same audio, and look at which terms fail, not only the average.
Is keyword accuracy the same as keyword spotting?
No. Keyword spotting is a detection task run on audio, often without a full transcript. Keyword accuracy evaluates a full transcript on how it handled target terms.
Should I use an NER model to find entities automatically?
It can speed up building references, but automatic tagging has its own errors, and running it on the hypothesis confuses recognition errors with tagging errors. Tag the reference once, check it by hand, and match against the hypothesis.
Why did recall rise but WER not fall after adding a word list?
Usually because precision fell. The system is now writing target terms in places they were not said, and those insertions and substitutions offset the gains. Check false alarms and WER on non-target words.
References#
- Del Rio, M. et al. (2021). Earnings-21: A Practical Benchmark for ASR in the Wild. Interspeech.
- Le, D. et al. (2021). Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion. Interspeech.
- Pundak, G., Sainath, T. N., Prabhavalkar, R., Kannan, A., & Zhao, D. (2018). Deep Context: End-to-End Contextual Speech Recognition. IEEE SLT.
- Morris, A. C., Maier, V., & Green, P. (2004). From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition. Interspeech.
- Szymański, P. et al. (2020). WER we are and WER we think we are. Findings of EMNLP.