Building your own ASR test set

A private ASR test set is a sample of your own audio with carefully produced reference transcripts, used to measure how speech recognition systems perform on the speech you actually process. It predicts production accuracy far better than any public benchmark, because it matches your microphones, speakers, languages and vocabulary. Building one well takes a few days of planning and a modest transcription budget; most of the value comes from sampling honestly and writing clear transcription rules.

Why public benchmarks are not enough#

Public benchmarks cover audiobooks, talks, parliament speeches, earnings calls and meetings. Almost none cover 8 kHz telephone audio, car cabins, clinic rooms, or the product names and accents of a particular customer base. They are also public, so any model may have been trained on them; see benchmark contamination. The ranking of systems on a benchmark often changes on domain audio, and the size of the gaps changes more.

The process#

1. Sample from production#

Take audio from the same pipeline the system will run on, after the same codecs and resampling. List the conditions that plausibly affect accuracy and sample so each is represented: telephone versus app audio, each language, main accent groups, quiet versus noisy environments, short commands versus long dictation. Keep a small, labelled share of known hard cases, but do not let them dominate, or the overall number will be pessimistic.

Avoid selection bias. Clips chosen because they were easy to find, or because someone remembered them as difficult, do not represent traffic. Draw randomly within each stratum, then check that no single speaker, customer or day accounts for a large share of the set.

2. Clear the rights#

Recorded speech of identifiable people is personal data under the GDPR. Confirm a lawful basis for using it for evaluation, restrict access, keep it in a controlled location, and redact or exclude clips containing health, payment or other sensitive data you do not need. See GDPR and voice data and PII redaction.

3. Transcribe from scratch#

References should be produced by people listening to the audio without a machine draft. Post-editing an ASR transcript is cheaper, but editors accept plausible machine choices they would not have made themselves, and the references drift towards that system's spelling, segmentation and formatting. The system that produced the draft then scores unfairly well.

4. Review#

Have a second transcriber review every file, and transcribe a subset of 5 to 10% independently. The WER between the two independent transcripts is your inter-annotator agreement: a floor below which differences between systems are hard to interpret.

5. Tag#

Mark the terms that matter, such as product names, medications, account numbers and amounts, so you can compute entity and keyword accuracy. Store metadata per clip: channel, language, accent group, noise level, duration and speaker count. Without metadata you cannot find out where a system fails.

6. Freeze and version#

Give the set a version number and do not change it silently. Fix errors in a new version and rescore every system on it. Keep it private.

Transcription guidelines#

A two-page guideline document prevents most scoring arguments. It should decide, with examples:

Transcribing verbatim and normalising at scoring time keeps options open. You can compute plain WER, WER without fillers, and semantic WER from the same references. You cannot recover a filler once it has been left out.

How big it needs to be#

Size is governed by what you want to detect. Errors are clustered: a hard clip has many errors, an easy one few. So the uncertainty of WER depends on the number of utterances and speakers as much as on the number of words. Bisani and Ney proposed bootstrap resampling over utterances to estimate confidence intervals, and the paired version is the standard way to compare two systems on the same set.

import random


def paired_bootstrap(errors_a, errors_b, ref_words, rounds=10_000, seed=0):
    rng = random.Random(seed)
    n = len(ref_words)
    diffs = []
    for _ in range(rounds):
        idx = [rng.randrange(n) for _ in range(n)]
        words = sum(ref_words[i] for i in idx)
        diffs.append((sum(errors_a[i] for i in idx) - sum(errors_b[i] for i in idx)) / words)
    diffs.sort()
    return diffs[int(0.025 * rounds)], diffs[int(0.975 * rounds)]

Each list holds one value per utterance: the error count for system A, for system B, and the reference word count. On a simulated set of 400 utterances, about 6,700 words, where one system's true error rate was 8% and the other's 7%, the 95% interval for the difference ran from −0.1 to +1.6 points. It included zero. A one-point gap on a set that size is suggestive, not conclusive.

As a working approach, start with a few hours of audio across the main strata, compute intervals, and grow the strata where intervals are too wide to support a decision. Per-stratum results need enough material in each stratum; a slice of twenty clips gives a number, not an answer.

Scoring and reporting#

• Run every system's output through the same normaliser, then compute pooled WER over the set.
• Report WER per stratum next to the total, with intervals.
• Report entity recall and precision, and exact-match rates for identifiers.
• Read a sample of errors by hand. Numbers tell you where to look; transcripts tell you what went wrong.
• Record the system version, settings and date of every run.

Common questions#

References#

1. Bisani, M., & Ney, H. (2004). Bootstrap estimates for confidence intervals in ASR performance evaluation. ICASSP.
2. Gillick, L., & Cox, S. J. (1989). Some statistical issues in the comparison of speech recognition algorithms. ICASSP.
3. Szymański, P. et al. (2020). WER we are and WER we think we are. Findings of EMNLP.
4. Likhomanenko, T. et al. (2021). Rethinking Evaluation in ASR: Are Our Models Robust Enough? Interspeech.
5. NIST Speech Recognition Scoring Toolkit (SCTK). GitHub.
6. Regulation (EU) 2016/679 (General Data Protection Regulation). EUR-Lex.