How to benchmark speech-to-text on your own audio

The only benchmark that tells you which speech-to-text model to use is one with audio that looks like your real production traffic, scored the same way for every provider. Published leaderboards and vendor WER figures are a starting point, but never fully reflect how your users actually speak. This guide covers how to build that test: what to measure, how to make the test set representative, why your reference transcripts may be less reliable than you think, and how much audio you really need before the numbers mean anything.

Start with what matters to your product

In speech recognition, “Which model is most accurate?” is a broad question with no useful answer on its own. Accurate on what? Ideally, get specific quickly. A few examples:

• A medical scribe cares about drug names and dosages
• A contact center cares about account numbers, product names, and callers on a bad mobile line
• A captioning product cares about speaker turns and proper nouns

Write down the two or three things that would break your product if the transcript got them wrong. Those will decide your test set and metric.

Which metric should you use?

There are a few metrics you can test on:

Word error rate (WER): For languages that separate words with spaces, which covers every European language. WER counts the substitutions, deletions, and insertions needed to turn the model’s output into the reference transcript, divided by the number of words in the reference.

Character error rate (CER): For languages without clear word boundaries, such as Chinese, Japanese, or Thai. The same calculation logic applies, but on character-level.

Both metrics are far from perfect. WER treats “the” and a misheard drug name as equally wrong, and does not measure if the meaning of the sentence survived. However, they’re still good enough to compare different models on the same data, which is what you’re trying to achieve.

If you care about something more narrow or specific, measure that directly. For example, if the risk in your product is product names or medical terms, track the error rate on that term list on its own. A model can win on overall WER and still perform poorly on the words your users actually care about and your product relies on.

Does your test audio sound like your production audio?

This is where most benchmarks go wrong. A clean read-speech dataset might contain the right language, but tells you very little about transcription performance on a noisy call recorded on a headset.

Ideally, match the test audio to production as closely as you can. That means:

• Words in full sentences: Domain-specific terminology spoken in a natural context, not read out as a list
• The same environment and background noise as your users: This can be a phone line, laptop mic, car, clinic, or open-plan office
• The same manner of speaking: Accents, speaking rate (speed), people talking over each other

A useful check is to play a clip from your test set next to a clip of real production audio. If you can too easily tell which is which, your test set isn’t representative. This is near impossible to get perfect, yet it’s important to take some steps to get your test audio in the right direction.

Should you use public datasets?

You can, but do so with care. If you use a public dataset, prioritize using niche ones. The popular, publicly available benchmark sets are already well-covered by every model provider, so strong numbers on them say more about how familiar the model is with that data than about how it will handle yours. Imagine you’re taking an exam at school or in university. If you take the test the first time, you might do okay. Take it a second time, and the exam becomes easier. Take it 20 times, and you’ll know the answers to every question. Testing on public benchmarks is no different, which is why a smaller dataset from your domain or recorded in conditions close to yours, is worth more than using a famous one.

To give you a visual idea of blind testing, compare our model with the reference transcripts below:

Real fully random samples from Gigaspeech-Cleaned where our model and the reference differ noticeably.

Compare blind, and shuffle the order

You’re human. Once you start reading transcripts, you will form opinions. Make your comparisons fair:

• Anonymize: Hide which provider produced which transcript
• Include the reference transcript as one of the anonymous options: You’ll be surprised how often it loses against the model output
• Shuffle the order for every sample: People tend to prefer whichever version they read first, so a fixed order quietly biases the result towards one model

This is a great way to test 2 models directly against each other, but also test a model against the reference.

Normalize the text, then check your normalization

Before scoring, both the reference transcript and the model output need to be normalized. This means aligning casing, punctuation, numbers (“twenty five” vs “25”), abbreviations, spelling variants. Without this, you run the risk of measuring formatting differences instead of recognition errors. The effect on WER can be large: the Whisper paper reports that its own text normalizer changed measured WER substantially on several datasets (Radford et al., 2022).

Treat normalization as an iterative process. Score the transcription first, then look at the most frequent errors and then fix the ones that are formatting discrepancies rather than incorrect recognition. Then score again. You’ll never get it perfect, so stop once the remaining top errors are real mistakes.

How reliable are your reference transcripts?

Less than most people assume. As Mark from our Research team puts it: “The reference usually has about as many mistakes as modern models.” Human transcribers often skip filler words, mishear names, normalize numbers inconsistently, and sometimes miss whole phrases.

That means some of the “errors” you might count actually belong to the reference, not the model. So before trusting any numbers or output, listen to examples and read the transcripts side by side with the audio. Twenty minutes of listening will probably tell you more about your test set than the aggregate score.

Can you compare your numbers with published ones?

Take published numbers with a giant grain of salt. A WER figure from a vendor page, a paper, or a leaderboard depends entirely on that team’s test set, their normalization, and the script they use to score. Two teams can run the same model on the same audio and report different numbers because one counts “25” vs “twenty five” as an error, while the other does not.

The only comparison you can trust is the one where every model went through your pipeline, on your audio, with your normalization.

Where should you look for errors?

Prioritize the worst audio samples, for example clips with a very high measured WER. Those are the ones that often reveal a broken reference, a language mix-up or a failure mode that matters. However, you should never only look at those. The worst 1% of clips is not representative of the other 99%, and if you only read failures you’ll come away with a skewed picture of both models. Make sure to read a random sample too.

How much audio do you need?

You need a lot more audio than you think. One hour of audio has huge error bars. At that size, a difference of 10% of WER between models can easily be noise.

You can estimate your own uncertainty without any statistics library. Take many random subsets of your test files, compute WER on each, and then look at how much the result moves. If two models’ ranges overlap heavily, you do not necessarily have a winner yet. This is a simple form of bootstrap resampling, which is a standard way to put confidence intervals on ASR results (Bisani and Ney, 2004).

One caveat: keep in mind that clips from the same speaker or call are correlated, so resample whole calls or speakers rather than only single clips (Liu et al., 2020).

Run the resampling for each model, on the same audio files. If the 95% ranges do not overlap, the difference is probably real. If they do, you need to add more audio. For context, the test set we use at Reson8 to evaluate our own models is more than 4,000 hours.

A checklist you can use to get started

• Write down the two or three things your product cannot afford to get wrong
• Measure WER for European languages, CER for languages without word boundaries, and a term-level error rate for the vocabulary that matters
• Build your test set from audio that sounds like your production audio (the “can I tell them apart?” test)
• If you use public data, pick niche datasets over popular ones
• Listen to samples yourself and check the reference before trusting any scores
• Compare transcripts blind, include the reference transcript in your comparison, and shuffle the order in which you compare
• Normalize text for every model the same way, and iterate on it
• Ignore published WER numbers when comparing providers
• Review the worst samples and a random sample as well
• Estimate error bars with random subsets, and add audio until the result is stable

How we approach this at Reson8

Every point made in this article is how our own Research team evaluates our own releases. They do this on a test set of more than 4,000 hours across 50+ languages. This is also why we’d rather you test us on your audio than take our word for it.

One thing worth including in your model assessment is the ability to steer on domain vocabulary. With Reson8, you can adapt the model to your terminology by uploading a text-only glossary or term list of up to 1 million tokens, applied in under 60 seconds with no fine-tuning or audio training data needed. That makes it quick and cheap to measure both the base model and the one adapted to the term list you care about. Try it, and see the difference for yourself.

Audio you send for testing is processed by Reson8 on our own GPU stack in the Netherlands. We never retain audio or transcripts, and never train on customer data. If you want help setting up a comparison on your own data, get in touch. If you’d like to get started with Reson8, sign up at console.reson8.dev.

Sources and further reading

• Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (the Whisper paper), including its section on text normalization and WER: arxiv.org/abs/2212.04356
• Bisani and Ney, Bootstrap estimates for confidence intervals in ASR performance evaluation, ICASSP 2004: rwth-aachen.de
• Liu et al., Statistical Testing on ASR Performance via Blockwise Bootstrap, Interspeech 2020: arxiv.org/abs/1912.09508
• Hugging Face Open ASR Leaderboard, for a sense of how public benchmark sets are scored: huggingface.co