Code-switching is when a speaker changes language within a conversation, often within a single sentence. It is normal in multilingual communities and in European workplaces, where English terms appear constantly in Dutch, German or Swedish speech. Speech recognisers that assume one language per utterance handle it poorly: they either force the foreign words into the wrong language or switch the whole transcript.
Key takeaways
- Switching can happen between sentences (inter-sentential) or inside one (intra-sentential). The second is much harder for ASR.
- Borrowing and switching sit on a continuum. "Meeting" in Dutch is a loanword; a full English clause is a switch. Both have to be transcribed in the right spelling.
- Per-utterance language identification cannot solve code-switching, because the utterance has no single language.
- Multilingual end-to-end models help, but mixed-language training data is scarce, so switched segments still see higher error rates.
- Evaluate on real mixed speech from your users. Monolingual test sets say nothing about switch points.
What code-switching looks like#
Linguists distinguish several patterns. Poplack's 1980 study of Puerto Rican Spanish-English speakers in New York, titled after an example sentence, "Sometimes I'll start a sentence in Spanish y termino en español", showed that switching follows grammatical constraints rather than being random. Myers-Scotton's Matrix Language Frame model describes mixed sentences as a frame from one language with content from another.
| Type | Example | Notes |
|---|---|---|
| Inter-sentential | "Dat klopt. Let's move on to the next item." | Switch at a sentence boundary |
| Intra-sentential | "We moeten die deadline echt halen voor de sprint review." | English nouns inside a Dutch frame |
| Tag switching | "Das ist fertig, right?" | A tag or discourse marker from another language |
| Loanwords, inflected | "Ik heb het gisteren gedeployed." | English stem with Dutch past participle morphology |
The last row shows why European business speech is hard. gedeployed is neither a Dutch word nor an English one. A model that treats it as Dutch may write gedeploid; one that treats it as English may drop the prefix. Similar forms occur across Germanic languages, such as German "gedownloadet", and in Polish, which attaches native case endings to English nouns.
Why code-switching breaks recognisers#
The one-language assumption#
Traditional pipelines pick a language, then load that language's acoustic model, lexicon and language model. Anything outside it is out of vocabulary. Many multilingual end-to-end models still condition on a single language token per segment, as Whisper does. They often cope with short, common English insertions but can switch the rest of the segment into the wrong language, or translate rather than transcribe.
Scarce training data#
Monolingual transcribed speech is plentiful for major European languages; transcribed mixed speech is not. Public code-switching corpora exist for only a handful of pairs, such as SEAME for Mandarin-English. Text data is also thin, because people code-switch far more in speech than in edited writing, so language models see few examples of mixed sentences.
Pronunciation drift#
Borrowed words are pronounced with the host language's sounds. A German speaker's "Update" and a Dutch speaker's "deadline" are not what an English acoustic model expects, and they are not native words either. See European English and accents and regional speech.
Spelling conventions#
Even when the words are recognised, the right spelling is disputed. Is it "gedeployed" or "ge-deployed"? "E-mailen" or "emailen"? References and hypotheses disagree for reasons unrelated to recognition, which inflates WER unless normalisation handles it.
Approaches#
- Multilingual models without a forced language. Models trained on many languages with a shared vocabulary can in principle output words from any of them at any point. They work best when trained on some mixed data.
- Bilingual models for a known pair. If callers mix Dutch and English, a model focused on those two, trained with mixed data, usually beats a hundred-language model.
- Word-level language tagging. Auxiliary LID at frame or token level can guide decoding or mark switch points in the output.
- Synthetic mixed text. Generating code-switched sentences from monolingual text, guided by linguistic constraints, helps language models see plausible mixes.
- Context biasing. Most intra-sentential switches in business speech are nouns: product names, job titles, technical terms. Supplying them as context through context biasing or text-only domain adaptation targets exactly those words.
Evaluating code-switched ASR#
Overall WER hides switching errors, because switched words are usually a small share of the total. Useful additions:
- Error rate on switched spans: WER computed only over words in the embedded language.
- Error rate around switch points: errors in a window of words either side of each switch, where models tend to fail.
- Language confusion rate: how often a word is output in the wrong language, including translated rather than transcribed.
For Mandarin-English, work on SEAME often reports a mixed error rate that counts Mandarin by characters and English by words, since the two languages use different natural units. The same idea applies whenever the languages being mixed segment text differently; see character error rate.
Common questions#
Is using English words in Dutch code-switching or borrowing?
Linguists debate the boundary. Established loanwords such as "computer" are borrowing; a spontaneous English phrase is switching. For ASR the distinction matters less than the result: both must be recognised and spelled as users expect.
Can I fix code-switching by running two models?
Rarely. You would need to know where the switch points are to route audio, which is the hard part. Models that handle both languages in one pass usually work better.
Does a multilingual model automatically handle code-switching?
No. Supporting many languages is not the same as supporting mixtures of them within a sentence. Test with real mixed speech before relying on it.
References#
- Poplack, S. (1980). Sometimes I'll start a sentence in Spanish y termino en español: toward a typology of code-switching. Linguistics, 18(7–8).
- Myers-Scotton, C. (1993). Duelling Languages: Grammatical Structure in Codeswitching. Oxford: Clarendon Press.
- Lyu, D.-C., Tan, T.-P., Chng, E. S., & Li, H. (2010). SEAME: a Mandarin-English Code-switching Speech Corpus in South-East Asia. Interspeech.
- Sitaram, S., Chandu, K. R., Rallabandi, S. K., & Black, A. W. (2019). A Survey of Code-switched Speech and Language Processing. arXiv:1904.00784.
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.