Confidence scores in speech recognition

A confidence score is the recogniser's estimate of how likely a word, or a whole utterance, is to be correct. It is useful for flagging words for human review, asking a caller to repeat something, or filtering unreliable transcripts. A score of 0.87 is only meaningful if the system is calibrated, meaning that words scored around 0.87 are in fact right about 87% of the time, and that has to be checked on your own audio.

Where confidence scores come from#

How a score is computed depends on the model architecture, and vendors rarely document it in detail.

Lattice posteriors#

Hybrid HMM systems produce a lattice of competing hypotheses. The posterior probability of a word is the share of total lattice probability that passes through it. Evermann and Woodland showed that these posteriors make good confidence scores, and they were the standard approach for many years. Jiang's 2005 survey covers this and older feature-based methods.

Token probabilities in end-to-end models#

CTC, RNN-T and attention models output a probability distribution at each step. A simple word confidence is the probability of the chosen token, or the minimum or mean over the subword tokens that form the word. These numbers are cheap but tend to be high even when the word is wrong, because models trained with cross-entropy and label smoothing are not naturally calibrated. Some systems use entropy of the distribution instead, which captures how many alternatives were competitive.

Dedicated confidence models#

A separate small model can be trained to predict whether each word is correct, using the recogniser's internal features as input and alignment with reference transcripts as labels. Li and colleagues described such a confidence estimation module for attention-based models, and reported better separation of correct and incorrect words than raw softmax probabilities.

Calibration#

A score is calibrated if, among all words scored near p, the fraction that are correct is close to p. Calibration is checked with a reliability diagram: bin words by confidence, compute accuracy in each bin, and compare with the diagonal.

A common summary is the expected calibration error:

where n_b is the number of words in bin b. Guo and colleagues showed in 2017 that modern deep networks are often poorly calibrated, and that a single temperature parameter fitted on held-out data, known as temperature scaling, fixes much of the problem. The same method applies to speech models.

Calibration is not the same as discrimination. A system can be perfectly calibrated and still useless, for example by giving every word the average accuracy as its score. What you want is scores that are both calibrated and spread out, so that low scores concentrate the errors. NIST's normalised cross entropy captures this: it rewards scores that carry information beyond the base rate of correct words.

import numpy as np

def expected_calibration_error(conf, correct, n_bins=10):
    conf = np.asarray(conf)
    correct = np.asarray(correct, dtype=float)
    bins = np.minimum((conf * n_bins).astype(int), n_bins - 1)
    ece = 0.0
    for b in range(n_bins):
        in_bin = bins == b
        if in_bin.any():
            ece += in_bin.mean() * abs(conf[in_bin].mean() - correct[in_bin].mean())
    return ece

The correct labels come from aligning hypothesis and reference, as for WER: a hypothesis word is correct if it matches, and incorrect if it is a substitution or insertion.

Using confidence scores#

Choosing a threshold#

Take a labelled sample of your own audio, score it, and for each candidate threshold compute two numbers: the fraction of words flagged, and the fraction of all errors that fall among the flagged words. Plot one against the other and choose the point your review budget can afford. The right threshold for medication names is not the right one for filler words, so thresholds per word class or per entity type often work better than one global value.

What confidence does not tell you#

• Deletions. Missed words have no score. A transcript with high confidence throughout can still have dropped half a sentence.
• Semantic importance. A 0.95 on the drug name and a 0.4 on "uh" says nothing about which matters. Combine confidence with what the word is.
• Systematic errors. If a model has learnt to spell a brand name wrong, it will spell it wrong confidently every time. Context biasing addresses that; confidence does not.
• Performance on shifted audio. Calibration fitted on clean speech often degrades on noisy, accented or telephone audio. Models are typically most overconfident exactly where they perform worst.

Common questions#

References#

1. Jiang, H. (2005). Confidence measures for speech recognition: A survey. Speech Communication, 45(4).
2. Evermann, G., & Woodland, P. C. (2000). Posterior probability decoding, confidence estimation and system combination. Proceedings of the NIST Speech Transcription Workshop.
3. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML.
4. Li, Q. et al. (2021). Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition. ICASSP.
5. Wessel, F., Schlüter, R., Macherey, K., & Ney, H. (2001). Confidence measures for large vocabulary continuous speech recognition. IEEE Transactions on Speech and Audio Processing, 9(3).