Spoken language identification (LID) detects which language is being spoken in a piece of audio. It lets a system pick the right recognition model, route a call to the right agent, or tag an archive by language without a human listening first. LID is reliable on several seconds of clear speech in well-separated languages, and much less reliable on short utterances, closely related languages and speakers who mix languages.
Key takeaways
- LID accuracy rises steeply with duration. A one-word answer such as "ja" is ambiguous between Dutch, German and Swedish.
- Closely related pairs are the hard cases: Spanish and Portuguese, Dutch and German, Swedish and Norwegian.
- Restricting the candidate set to the languages you actually serve is the single most effective way to improve LID.
- Many multilingual ASR models, including Whisper, identify the language as part of decoding, usually once per 30-second window.
- Per-utterance LID assumes one language per utterance. Mixed-language speech needs code-switching support instead.
Where LID is used#
- Model routing. Detect the language, then send audio to a language-specific or language-conditioned recogniser. A wrong guess here is catastrophic: an English model transcribing Polish produces fluent nonsense, not a few errors.
- Contact centres. Route callers to an agent who speaks their language, or switch a multilingual voice agent into the caller's language.
- Media and archives. Tag hours of recordings by language for search and subtitling.
- Analytics. Measure which languages customers use, by region and channel.
How LID works#
Phonotactic systems#
Early systems exploited the fact that languages differ in which sound sequences they allow. A phone recogniser, sometimes trained on a single language, turned audio into phone strings, and a separate n-gram model per language scored how typical those strings were. Zissman's 1996 comparison of four such approaches on telephone speech was a reference point for years, and phone recognition followed by language modelling (PRLM) was a standard baseline.
Embeddings#
Acoustic approaches then took over, first with i-vectors and later with neural embeddings. Snyder and colleagues showed in 2018 that x-vectors, originally designed for speaker recognition, work well for language recognition when trained on language labels. The recipe is the same as for speaker embeddings: an encoder, statistics pooling over time and a classifier, with the target being the language rather than the person.
LID inside multilingual ASR#
Large multilingual recognisers often do LID implicitly. Whisper predicts a language token at the start of each 30-second window, and that token conditions the transcription that follows. This is convenient, but it means one decision per window, and a wrong first guess affects the whole window.
import whisper
model = whisper.load_model("small")
audio = whisper.pad_or_trim(whisper.load_audio("call.wav"))
mel = whisper.log_mel_spectrogram(audio, n_mels=model.dims.n_mels).to(model.device)
_, probs = model.detect_language(mel)
print(sorted(probs.items(), key=lambda kv: -kv[1])[:3])
What makes LID hard#
| Condition | Example | Why it fails |
|---|---|---|
| Very short audio | "Ja", "OK", "Hallo" | Too little evidence; the word exists in several languages |
| Related languages | Spanish and Portuguese, Dutch and German | Shared vocabulary and similar sound inventories |
| Accented speech | French spoken with a strong Polish accent | Accent cues pull towards the speaker's first language |
| Mixed language | Dutch with English business terms | No single correct label for the utterance |
| Names and numbers | An address or order number read out | Proper nouns and digits carry little language signal |
| Telephone audio | 8 kHz calls | Less spectral detail than wideband training data |
Accent and language interact in both directions. LID models can be confused by accent, and accent identification is sometimes treated as a finer-grained version of the same task. See accents and regional speech and European English.
Using LID in production#
Constrain the candidate set#
If your service handles five languages, score only those five. Open-ended LID across a hundred languages spends probability on languages your callers never use, and a Dutch caller misclassified as Afrikaans is a routing failure you could have ruled out.
Use priors from context#
Phone number country code, account language, website locale and the language of the previous call are strong priors. Combine them with the acoustic score rather than relying on audio alone.
Decide late, or revise#
Accuracy rises with duration, so a system that waits a few seconds before committing makes fewer errors. In a live conversation that costs latency; a common compromise is to start in the most likely language and allow one switch once enough speech has been heard.
Treat low confidence as a signal#
When the top two languages are close, asking "Would you prefer English or Dutch?" is cheaper than transcribing a call in the wrong language. See confidence scores for how to set thresholds.
Evaluating LID#
Closed-set LID is usually reported as accuracy or error rate, ideally broken down by language pair in a confusion matrix, since errors concentrate on related languages. The NIST Language Recognition Evaluations use a detection cost metric (Cavg) that treats each language as a detection task. Public datasets include VoxLingua107, built from YouTube audio in 107 languages, and FLEURS, which covers around a hundred languages with parallel read sentences.
Common questions#
How much audio does LID need?
There is no single threshold, but errors fall sharply over the first few seconds of speech. For closely related languages you may need considerably more, especially on telephone audio.
Can LID handle speakers who switch languages?
Per-utterance LID cannot, because it outputs one label. Frame-level or word-level LID can mark switch points, and models trained on mixed speech can transcribe without an explicit decision. See code-switching.
Should I let the ASR model detect the language or set it explicitly?
Set it explicitly when you know it. Automatic detection is a fallback for when you do not, and it adds an error mode that a fixed setting avoids.
References#
- Zissman, M. A. (1996). Comparison of Four Approaches to Automatic Language Identification of Telephone Speech. IEEE Transactions on Speech and Audio Processing, 4(1).
- Snyder, D., Garcia-Romero, D., McCree, A., Sell, G., Povey, D., & Khudanpur, S. (2018). Spoken Language Recognition using X-vectors. Odyssey: The Speaker and Language Recognition Workshop.
- Valk, J., & Alumäe, T. (2021). VoxLingua107: a Dataset for Spoken Language Recognition. IEEE Spoken Language Technology Workshop (SLT).
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.
- Conneau, A. et al. (2022). FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. arXiv:2205.12446.