Character error rate (CER)

Character error rate (CER) is the character-level counterpart of word error rate: the number of character substitutions, deletions and insertions needed to turn a transcript into its reference, divided by the reference length in characters. It is the standard metric for languages written without spaces, such as Mandarin and Japanese, and a useful second metric for languages where one misplaced word boundary would otherwise cost several word errors.

Definition#

The calculation is the same Levenshtein edit distance that underlies WER, applied to a sequence of characters instead of a sequence of words:

Here S, D and I are character substitutions, deletions and insertions, and N is the number of characters in the reference. Like WER, CER can exceed 100% if the hypothesis contains many inserted characters, although in practice that only happens with badly hallucinated output.

The unit changes what an error means. A word error says "this word is wrong". A character error says "this letter is wrong", and a word with one wrong letter costs one character error out of the several characters in that word. CER therefore rewards near misses that WER treats as total failures.

When words are the wrong unit#

Languages without spaces#

Written Mandarin, Cantonese, Japanese and Thai do not separate words with spaces. Any word-level metric first needs a word segmenter, and segmenters disagree with each other. Two systems could produce identical characters and receive different WER depending on the tool used to split them. Scoring characters removes the segmenter from the evaluation. Mandarin corpora such as AISHELL-1 report CER for this reason.

Compounding languages#

Dutch, German, Swedish and, to a lesser degree, Polish build long words by joining shorter ones. The official spelling of a compound is sometimes debated even among native writers. When a model writes ziekenhuis opname for ziekenhuisopname (hospital admission), WER scores one substitution and one insertion: two errors for one reference word. The recognition was correct; only the space was wrong. See compound words and morphology for why this is so common.

Diacritics and spelling variants#

A model that writes Lodz for Łódź has the right word with three wrong characters. WER counts one error; CER counts three out of four. Here CER is the harsher metric. Which behaviour you want depends on whether a missing diacritic matters downstream. For search it often does not; for a legal name or an address it does.

Computing CER#

A self-contained implementation is short. The function below works on any sequence, so the same code gives WER on word lists and CER on strings:

def edit_distance(ref, hyp):
    prev = list(range(len(hyp) + 1))
    for i, r in enumerate(ref, 1):
        curr = [i]
        for j, h in enumerate(hyp, 1):
            curr.append(min(prev[j] + 1, curr[j - 1] + 1, prev[j - 1] + (r != h)))
        prev = curr
    return prev[-1]


def error_rate(ref, hyp):
    return edit_distance(ref, hyp) / len(ref)


ref = "de patiënt wacht op ziekenhuisopname"
hyp = "de patient wacht op ziekenhuis opname"

print(f"WER {error_rate(ref.split(), hyp.split()):.1%}")   # 60.0%
print(f"CER {error_rate(ref, hyp):.1%}")                   # 5.6%
print(f"CER no spaces {error_rate(ref.replace(' ', ''), hyp.replace(' ', '')):.1%}")  # 3.1%

One sentence, three plausible numbers. The WER of 60% comes from a missing diaeresis and a split compound. The CER of 5.6% counts the same two slips against 36 characters. Dropping spaces removes the segmentation error and leaves only the diacritic. The jiwer package also provides a cer function if you prefer a library.

Choices that change the score#

Numbers deserve particular care. Character counts inflate the cost of a formatting difference: "€1,250" against "twelve hundred and fifty euros" is a large character distance for a transcript that may be entirely correct. Run the same text normalisation you would use for WER before measuring characters.

Interpreting CER alongside WER#

For alphabetic languages, CER is almost always lower than WER on the same data, because most word errors involve only a few wrong letters within a longer word. The ratio varies with language, word length and error type, so there is no fixed conversion. A system with 8% WER and 3% CER is not better or worse than one quoted at 5% WER on another set; the numbers answer different questions.

Reading the two together is informative. If WER is high but CER is low, the errors are mostly near misses: spelling variants, diacritics, split or joined compounds. That pattern suggests a normalisation or segmentation issue rather than a recognition problem. If both are high, the model is mishearing the audio, and the fix lies in the model, the audio or domain adaptation.

CER is less suited to identifiers. For an order number or licence plate, one wrong character makes the whole code unusable, so a character-level average understates the damage. There, exact-match rates on the identifier as a whole are a better measure; see entity and keyword accuracy and structured identifiers.

Common questions#

References#

1. Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8).
2. Bu, H., Du, J., Na, X., Wu, B., & Zheng, H. (2017). AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline. O-COCOSDA.
3. Conneau, A. et al. (2023). FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. IEEE SLT 2022.
4. The Unicode Consortium. Unicode Standard Annex #15: Unicode Normalization Forms. unicode.org.
5. jiwer: evaluate speech recognition with WER, MER, WIL and CER. GitHub.