A multilingual voice agent has to recognise, reason and speak in the caller's language, and sometimes follow the caller when they change language mid-call. The language decision touches every part of the pipeline: which STT language or model to use, which prompt and knowledge the LLM gets, which TTS voice replies, and even how the turn detector interprets pauses and fillers. The hardest cases are not choosing between two languages but short answers, mixed-language speech and regional variants.
Key takeaways
- Decide the language as early and as cheaply as possible: from the dialled number, the account, or the first full sentence, not from "ja".
- Language affects more than transcription. Numbers, dates, fillers, prosody and politeness conventions all differ, and each component has to handle them.
- Keep one voice identity across languages where you can. A caller who switches to French should not suddenly hear a different person.
- Code-switching is normal in Europe: English product names in Dutch sentences, French terms in Flemish ones. Test for it explicitly.
- Evaluate each language separately, on its own audio. An average across languages hides the weakest one.
Three patterns of multilingual calls#
Most real deployments meet some mix of these:
- Language per call. The caller uses one language throughout, and the agent must identify it. This covers most contact-centre traffic in multilingual countries such as Belgium or Switzerland.
- Switch mid-call. The caller starts in one language and moves to another, often because they started in the one they thought the agent expected. "Do you speak German?" is a common second turn.
- Mixing within a turn. The caller speaks one language with words from another: ik wil mijn subscription cancellen. This is code-switching, and a recogniser locked to one language will mangle the other language's words.
Routing the language through the pipeline#
Deciding the language#
Use context before audio. The dialled number, a country-specific IVR option, the caller's account profile or the website language often settle the question at no cost and with no error. When they do not, spoken language identification on the first turn is the usual fallback.
Language identification is much less reliable on short audio. A one-word answer carries little evidence, and some words are shared across languages with different meanings.
Recognition#
There are two broad options. A multilingual model transcribes several languages and reports which one it heard, which handles switches and some code-switching without re-routing. Per-language models, selected by the orchestrator, can be stronger in each language but need a correct language decision first and a way to re-route when the caller switches. Either way, give the recogniser the language when you know it: forcing the language removes one class of error entirely.
Language model#
Modern LLMs are multilingual, but quality varies by language, and prompts written in English tend to leak English phrasing into other languages. Keep a reviewed system prompt per language, or at least explicit instructions on reply language, register and forms of address. French and German callers may expect vous and Sie from a company; Dutch and Swedish service conversations are often less formal.
Speech synthesis#
Choose a TTS voice that exists in every language you support, or a multilingual voice that keeps the same timbre. Check pronunciation of brand names, place names and numbers in each language separately, and maintain a pronunciation lexicon per language.
Language changes more than words#
| Area | Example of variation | Component affected |
|---|---|---|
| Numbers | Dutch and German say units before tens: "vierennegentig", "vierundneunzig" (four-and-ninety) | STT, ITN, TTS |
| Regional variants | France "quatre-vingt-dix", Belgium and Switzerland "nonante" for 90 | STT, ITN, TTS |
| Dates | Day-month order, spoken ordinals, month names | ITN, tool arguments |
| Fillers | "euh", "äh", "eh", "yyy" | Turn detection |
| Spelling alphabets | Different national conventions for spelling names aloud | STT, prompt |
| Compounds | German and Dutch join words into long compounds | STT, evaluation (see compounds) |
Turn-taking itself varies. Stivers et al. found that the timing of responses differs measurably between languages, even though all avoid long gaps and overlap. A turn detector tuned only on English conversation can misjudge pauses in other languages, and filler words it has never seen may be taken as silence.
Evaluating a multilingual agent#
- Per-language test sets from your real channel, with native-speaker references. Public sets such as FLEURS cover many languages but consist of read speech, not calls. See building your own ASR test set.
- Language identification accuracy on the first turn, including short answers.
- Switch scenarios: caller changes language on turn two, on turn six, and back again.
- Code-switched speech: sentences with embedded English or other-language terms typical for your domain.
- Regional variants: Flemish and Netherlands Dutch, Belgian and Swiss French, Austrian and Swiss German, European and Brazilian Portuguese.
- Native review of replies: grammar, register and naturalness of the agent's speech, which automated metrics measure poorly.
Report every metric per language. An average WER across nine languages can look fine while one language is unusable. The testing voice agents article covers simulation methods that extend naturally to multilingual scenarios.
How Reson8 approaches this#
Resonant-1 covers nine European languages in depth: Dutch, English, French, German, Italian, Polish, Portuguese, Spanish and Swedish. That is far fewer than models offering around a hundred languages, by design. On FLEURS it averaged 4.22% WER excluding Swedish (Dutch 4.88%, French 4.71%, German 3.83%, Spanish 2.67%, Polish 4.56%, English 4.69%), with Swedish at 7.38%. The Turns endpoint brings streaming recognition, eager end-of-turn prediction and barge-in detection to voice agents; see the documentation for language configuration. Full results are in the benchmarks post.
Common questions#
Should the agent ask the caller which language they want?
Only if context cannot settle it. A spoken menu adds a turn to every call. Detecting the language from the first sentence, and confirming only when uncertain, is usually smoother.
Can one agent really support many languages?
Technically yes. In practice each language needs its own testing, prompt review and pronunciation checks, so the cost scales with the number of languages you support well.
How do I handle English words inside another language?
Use a recogniser that handles code-switching, and supply domain terms through context biasing so product names are recognised whatever language surrounds them. See custom vocabulary and context biasing.
References#
- Conneau, A. et al. (2022). FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. arXiv:2205.12446.
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.
- Pratap, V. et al. (2023). Scaling Speech Technology to 1,000+ Languages. arXiv:2305.13516.
- Sitaram, S. et al. (2019). A Survey of Code-switched Speech and Language Processing. arXiv:1904.00784.
- Stivers, T. et al. (2009). Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106(26).
- Phillips, A., & Davis, M. (2009). Tags for Identifying Languages. BCP 47 / RFC 5646, IETF.