Multilingual speech recognition uses one model to transcribe many languages. Sharing a model lets languages with little data borrow from languages with a lot, simplifies deployment, and handles speakers who switch language mid-conversation. The costs are shared capacity, which can dilute accuracy per language as more are added, and a new class of error: output in the wrong language. Whether one model or several is better depends on how many languages you need, and how deeply.
Key takeaways
- Speech sounds overlap across languages, so a shared encoder learns representations that transfer, most of all to low-resource languages.
- At a fixed model size, adding languages eventually reduces accuracy per language. More languages need more capacity.
- Language confusion, where a model transcribes or translates into the wrong language, is specific to multilingual systems.
- Language detection is unreliable on short utterances. If you know the language, say so.
- Evaluate per language and on code-switched speech; an average across languages hides the weak ones.
Two designs#
Before large multilingual models, the standard approach was a separate recogniser per language, with a language identification step in front to route audio. Today most general-purpose systems use one multilingual model, often guided by a language token.
| Per-language models | One multilingual model | |
|---|---|---|
| Low-resource languages | Limited by their own data | Benefit from transfer |
| Accuracy per language | Full capacity for each language | Capacity shared; can dilute as languages are added |
| Code-switching | Poor; one model per segment | Possible within one utterance |
| Operations | Many models to serve and update | One model to serve |
| Language errors | Routing errors send audio to the wrong model | Output can drift into the wrong language |
How multilingual models share knowledge#
Human languages reuse a limited set of sounds. A vowel close to /a/, a plosive like /t/, a nasal like /m/ occur in almost every European language. A shared acoustic encoder learns these once and applies them everywhere. Self-supervised models such as XLSR and XLS-R showed that pretraining on untranscribed audio from many languages produces representations that transfer strongly to languages with little labelled data.
On the output side, multilingual models use a shared subword vocabulary. Related languages share many subwords: Dutch and German, Spanish and Portuguese, Swedish and the other Scandinavian languages. That sharing helps with cognates and loanwords and occasionally causes spelling from one language to leak into another.
Most multilingual decoders also carry a language token. Whisper, for example, predicts a language token at the start of each segment, and the caller can set it to force a language. With the token fixed, the model is steered toward that language's vocabulary and spelling.
The capacity trade-off#
Transfer is not free. Work on multilingual text models described a curse of multilinguality: at a fixed model size, adding languages first helps low-resource languages, then begins to reduce accuracy across the board as each language's share of capacity shrinks. Large multilingual ASR studies have reported the same pattern and found that increasing model size recovers much of the loss.
The practical reading is that a model covering a hundred languages at a given size is making a different trade than a model of similar size covering ten. Its headline languages may be excellent while its tail languages are much weaker. Published per-language tables, such as those on FLEURS, usually show this directly.
Failure modes specific to multilingual systems#
- Wrong-language output. The model recognises the audio as a different language and transcribes it phonetically in that language's spelling, or translates it.
- Detection on short audio. "Ja", "no", "OK" and names are common across languages. Language identification on a one-second utterance is close to guessing, which matters in voice agents where turns are short.
- Spelling bleed. English spelling applied to Dutch or German loanwords, or Spanish accents dropped in Portuguese.
- Code-switching. A Dutch sentence with an English phrase in the middle: we moeten die deadline echt halen vs we moeten die dead line echt halen. See code-switching.
- Formatting by the wrong locale. Numbers written with a decimal point in a French transcript. See numbers and alphanumerics.
Evaluating a multilingual system#
- Per-language WER. Report each language separately. An unweighted average hides weak languages; a weighted average reflects your traffic. Report both.
- Language-specific normalisation. Use a normaliser appropriate to each language; an English normaliser applied to Polish will distort the score. See word error rate.
- Language identification accuracy. Measure how often the detected language is right, at the utterance lengths you actually see.
- Code-switched set. Include utterances that mix languages the way your users do.
- Forced versus detected. Run the same test with the language fixed and detected. The difference is the cost of detection errors.
How Reson8 approaches this#
Reson8 covers nine European languages in depth: Dutch, English, French, German, Italian, Polish, Portuguese, Spanish and Swedish. On FLEURS, Resonant-1 averaged 4.22% WER across these languages excluding Swedish, and led in French, Dutch, Spanish and Polish; Swedish, at 7.38%, is reported separately. Covering nine languages rather than a hundred means languages outside those nine are not supported. Per-language results are on the benchmarks page, and language settings are described in the documentation.
Common questions#
Should I use automatic language detection?
Only when you cannot know the language in advance. Detection adds a source of error, especially on short turns. Where you know the language, set it.
Can one model handle a conversation in two languages?
A multilingual model can, if it was trained or tested for code-switching. Check with recordings of your own users, since switching patterns differ between communities.
Is a monolingual model always more accurate?
Not always. For high-resource languages a dedicated model of the same size can be slightly better. For lower-resource languages, transfer from a multilingual model often outweighs the capacity cost.
References#
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.
- Conneau, A. et al. (2020). Unsupervised Cross-lingual Representation Learning for Speech Recognition (XLSR). arXiv:2006.13979.
- Babu, A. et al. (2021). XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. arXiv:2111.09296.
- Conneau, A. et al. (2020). Unsupervised Cross-lingual Representation Learning at Scale (XLM-R). ACL.
- Pratap, V. et al. (2020). Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters. Interspeech.
- Pratap, V. et al. (2023). Scaling Speech Technology to 1,000+ Languages. arXiv:2305.13516.
- Conneau, A. et al. (2022). FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. IEEE SLT.