Speech recognition accuracy varies with who is speaking. Models trained mostly on standard, broadcast-style speech make more errors on regional accents, dialects and non-native speakers, sometimes by a wide margin. The gap is a data problem more than an architecture problem: models are good at the varieties they heard in training. Measuring accuracy per accent group, rather than as one average, is the first step to closing it.
Key takeaways
- Accent affects vowels, consonants, rhythm and speaking rate. Dialect adds vocabulary and grammar on top.
- Published studies have found large accuracy gaps between speaker groups in commercial systems, in English and in Dutch.
- An average WER can look good while one group of your users gets a much worse service.
- Diverse training data and self-supervised pretraining help most. Text adaptation helps with regional vocabulary, not pronunciation.
- Test with enough distinct speakers per group; many utterances from a few people measure those people, not the accent.
What varies between speakers#
An accent is a pattern of pronunciation. A dialect is a fuller variety of a language with its own words and sometimes its own grammar. Both matter for recognition, in different ways.
- Vowels. The same written vowel can be realised very differently across regions. A model that has learned one realisation may map another to a different word.
- Consonants. Dutch g is a hard, rasping sound in much of the Netherlands and a soft one in the south and in Flanders. Rolled, tapped and uvular r all occur within French, German and Dutch.
- Prosody and rate. Rhythm, stress placement and speaking rate differ between varieties, and fast speech merges syllables that slow speech keeps apart.
- Vocabulary. Belgian and Swiss French say septante and nonante where France says soixante-dix and quatre-vingt-dix. Flemish speakers call a mobile phone a gsm.
- Diglossia. Swiss German is spoken in everyday life but Standard German is normally written, so transcribing Swiss German into standard spelling is partly a translation task.
Non-native (L2) speech combines several of these. A speaker carries vowels, consonants and stress patterns from their first language into the second. See European English for the most common case in European workplaces.
Why models struggle#
Recognisers learn the varieties in their training data. Public speech corpora skew toward read speech, broadcast media and parliamentary recordings, which in turn skew toward standard varieties. Volunteer-recorded datasets such as Common Voice broaden the speaker pool but depend on who volunteers.
The effect has been measured. Koenecke and colleagues (2020) tested five commercial English systems on interviews and found an average WER of 0.35 for Black speakers against 0.19 for white speakers. Feng and colleagues (2021) evaluated a state-of-the-art Dutch system on the Spoken Dutch Corpus and found higher error rates for non-native speakers, children and older speakers, and differences by regional accent. Tatman (2017) reported dialect-related differences in automatic captions.
Accent errors are mostly substitutions of one valid word for another, so the transcript still reads fluently and the error is hard to spot. Regional place names and local vocabulary add a second layer, because they are rare in training text as well as in training audio. See rare words and proper nouns for that side of the problem.
Measuring the gap#
A single test set average hides who the errors fall on. Slice evaluation reports WER separately for each speaker group.
Practical rules for building the slices, alongside the general guidance in building your own ASR test set:
- Count speakers, not hours. Speaker variation is large. Ten hours from three speakers tells you about three people.
- Match your users. If your callers are mostly from Limburg and Antwerp, a test set of Randstad speech will overestimate accuracy.
- Collect metadata with consent. Accent group and first language are personal data. Record them only with a lawful basis and clear purpose; see GDPR and voice data.
- Normalise regional spelling fairly. If a speaker says septante, decide in advance whether "70" or "soixante-dix" is correct output. See inverse text normalisation.
Variation in European languages#
| Language | Major varieties | Typical recognition issue |
|---|---|---|
| Dutch | Netherlands, Flanders, Suriname | Soft and hard g; Flemish vocabulary. See Dutch. |
| German | Germany, Austria, Switzerland | Swiss German spoken, Standard German written. See German. |
| French | France, Belgium, Switzerland, Canada | Regional number words; vowel differences. See French. |
| Spanish | Peninsular, Latin American | Seseo and ceceo; vocabulary. See Spanish. |
| Portuguese | European, Brazilian | Vowel reduction in European Portuguese. See Portuguese. |
| Italian | Regional varieties | Strong regional accents alongside local dialects. See Italian. |
| Swedish | Sweden, Finland Swedish | Pitch accent and regional prosody. See Swedish. |
Improving accuracy across accents#
Broader training data#
The most effective fix is audio from the varieties that underperform. This is the domain of the model provider; as a buyer, ask which varieties are represented and ask for sliced results.
Self-supervised pretraining#
Models such as wav2vec 2.0 learn speech representations from untranscribed audio. Because untranscribed audio is far easier to collect across regions than transcribed audio, this is a practical way to widen the range of voices a model has heard.
Targeted fine-tuning#
Where you own audio from a specific group, fine-tuning on it can close a specific gap. It needs transcribed data and a maintained model version. See fine-tuning vs context biasing.
Text adaptation for regional vocabulary#
Text-only adaptation does not change what the model hears, but regional place names, local business names and dialect words are a vocabulary problem, and text fixes those.
Design for confirmation#
In a voice agent, confirming key values by read-back protects every speaker group, and helps most where recognition is weakest.
How Reson8 approaches this#
Reson8 covers nine European languages in depth: Dutch, English, French, German, Italian, Polish, Portuguese, Spanish and Swedish. Published FLEURS results for Resonant-1 are listed on the benchmarks page. FLEURS is read speech and does not measure regional accent gaps, so evaluate on recordings of your own speakers. Regional names and terms can be supplied through text-only domain adaptation; see the documentation.
Common questions#
Can a model be accent-neutral?
Not fully. Every model reflects its training data. The practical goal is a small, measured gap across the groups you serve.
Should I choose the language variant, such as Belgian Dutch, explicitly?
If the system offers it and your users are homogeneous, test both settings. For mixed users, a single model that handles all varieties is simpler to operate.
Does noise make accent gaps worse?
Often, yes. Degraded audio removes cues the model uses to cope with unfamiliar pronunciation. Test accents on your real channel, such as telephony.
How many speakers per group do I need?
There is no fixed number, but dozens of distinct speakers per group give far more reliable estimates than a handful.
References#
- Koenecke, A. et al. (2020). Racial Disparities in Automated Speech Recognition. Proceedings of the National Academy of Sciences, 117(14).
- Feng, S., Kudina, O., Halpern, B. M., & Scharenborg, O. (2021). Quantifying Bias in Automatic Speech Recognition. arXiv:2103.15122.
- Tatman, R. (2017). Gender and Dialect Bias in YouTube's Automatic Captions. Proceedings of the First ACL Workshop on Ethics in Natural Language Processing.
- Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. NeurIPS.
- Ardila, R. et al. (2020). Common Voice: A Massively-Multilingual Speech Corpus. LREC.
- Weinberger, S. Speech Accent Archive. George Mason University.