Speech recognition works out what was said; voice recognition works out who said it. Speech recognition produces text and tries to ignore differences between speakers. Voice recognition, properly called speaker recognition, produces an identity or a match score and tries to ignore what was said. The two use similar audio features and neural networks, but they are trained for opposite goals and carry very different privacy obligations.
Key takeaways
- Speech recognition (ASR) outputs words. Speaker recognition outputs an identity, a match decision or speaker labels.
- Consumer products use "voice recognition" loosely for both, which causes confusion in requirements and contracts.
- Speaker recognition splits into verification (is this person who they claim?), identification (which known person is this?) and diarisation (who spoke when, without names).
- A voiceprint used to identify someone is biometric data. Under GDPR that makes it special category data, with stricter rules than a transcript.
- Many products need both: a transcript attributed to speakers, or a phone line that verifies the caller then transcribes the call.
Two different questions#
Speech recognition maps audio to the sequence of words spoken. A good speech recogniser gives the same transcript whether a sentence is spoken by a child in Groningen or a man in Glasgow. Differences between voices are noise it must learn to discount.
Speaker recognition maps audio to information about the speaker. A good speaker recogniser gives the same answer whether the person says their name, reads a number or talks about the weather. The words are noise it must discount. This is why the two tasks, though both built on speech audio, are trained on different labels and optimised against different metrics.
The speaker recognition family#
"Voice recognition" in the technical sense covers several related tasks, which the diarisation vs identification vs verification article treats in depth.
| Task | Question | Output | Example |
|---|---|---|---|
| Speaker verification | Is this the person they claim to be? | Accept or reject, with a score | Bank phone line voice authentication |
| Speaker identification | Which enrolled person is this? | A name from a known set, or "unknown" | Smart speaker recognising household members |
| Speaker diarisation | Who spoke when? | Anonymous labels on time segments | Meeting transcript with Speaker 1 and Speaker 2 |
| Language identification | Which language is this? | A language code | Routing a call to a Dutch or French model |
Verification can be text-dependent, where the user says a fixed passphrase, or text-independent, where any speech will do. Text-dependent systems need less audio for the same accuracy; text-independent systems can verify passively during a normal conversation.
How speaker recognition works#
Speaker recognition starts from the same log-mel or MFCC features as speech recognition. The difference is in what the network is trained to predict. Instead of text, it is trained to classify thousands of training speakers. After training, the classification layer is discarded and an intermediate layer is used as a speaker embedding: a vector, typically a few hundred numbers, that places voices of the same person close together.
The field moved through the same eras as ASR. Gaussian mixture models with a universal background model were standard around 2000. I-vectors, a compact factor-analysis representation, followed around 2010. Neural x-vectors, introduced by Snyder and colleagues in 2018, and later architectures such as ECAPA-TDNN now dominate. Public datasets such as VoxCeleb, built from celebrity interview videos, made large-scale training and comparison possible.
To verify, the system compares an embedding from the new audio with one stored at enrolment, usually with cosine similarity, and accepts if the score passes a threshold. The threshold trades false acceptances against false rejections; the equal error rate is the point where the two are equal and is the usual headline metric.
Where the two differ in practice#
| Speech recognition | Speaker recognition | |
|---|---|---|
| Output | Text | Identity, score or labels |
| Training labels | Transcripts | Speaker identities |
| Main metric | Word error rate | Equal error rate, diarisation error rate |
| Needs enrolment | No | For verification and identification, yes |
| Speaker variation | A nuisance to ignore | The signal |
| GDPR status of output | Personal data if it identifies someone | Biometric data when used to identify |
Privacy and regulation#
The legal difference is the most consequential. A transcript is personal data when it relates to an identifiable person, and falls under GDPR in the ordinary way. A voiceprint processed to uniquely identify someone is biometric data, which Article 9 of the GDPR treats as special category data: processing is prohibited unless a specific condition applies, most often explicit consent. See GDPR and voice data and voice biometrics and consent.
The EU AI Act adds further rules for biometric systems, including classifying certain remote biometric identification systems as high-risk and restricting emotion recognition in workplaces and schools. Transcription on its own is not a biometric use. See the EU AI Act and speech recognition.
When products need both#
Many systems combine the two. A meeting assistant runs ASR for the words and diarisation to split them by speaker, then optionally maps speaker labels to participant names. A bank's phone line may verify the caller's voice, then transcribe the rest of the call for records. A car assistant may identify which occupant is speaking to load their preferences, then recognise the command.
Combining them introduces its own errors. When diarisation assigns a sentence to the wrong speaker, the words are right but the transcript is wrong: Customer: I want to cancel becomes Agent: I want to cancel. Evaluate attribution separately from word accuracy.
Common questions#
Is "voice recognition" ever used to mean speech recognition?
Often, in everyday use and product marketing: "voice recognition software" usually means dictation. In technical and legal writing, keep the distinction: speech recognition for content, speaker or voice recognition for identity.
Can a speech recognition system identify me?
Not by design. It is trained to ignore who is speaking. The transcript may still identify you through its content, such as your name or account number, which is why transcripts are treated as personal data.
Is speaker diarisation biometric processing?
Diarisation groups speech by voice similarity within one recording without linking it to a known identity. Whether it counts as biometric processing depends on how it is used; mapping clusters to named, enrolled people moves it closer to identification. Take legal advice for your specific use.
Does voice verification work on the phone?
Yes, and it is deployed in banking and telecoms, though 8 kHz audio and codec compression reduce accuracy compared with wideband recordings, and synthetic voice attacks are a growing concern.
References#
- Hansen, J. H. L., & Hasan, T. (2015). Speaker recognition by machines and humans: a tutorial review. IEEE Signal Processing Magazine, 32(6).
- Reynolds, D. A., Quatieri, T. F., & Dunn, R. B. (2000). Speaker verification using adapted Gaussian mixture models. Digital Signal Processing, 10(1–3).
- Dehak, N., Kenny, P., Dehak, R., Dumouchel, P., & Ouellet, P. (2011). Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing, 19(4).
- Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. (2018). X-vectors: robust DNN embeddings for speaker recognition. ICASSP.
- Nagrani, A., Chung, J. S., & Zisserman, A. (2017). VoxCeleb: a large-scale speaker identification dataset. Interspeech.
- Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 4(14) and 9. EUR-Lex.