Speaker diarization, speaker identification and speaker verification all analyse voices, but they answer different questions. Diarization asks who spoke when, using anonymous labels. Identification asks which known person is speaking. Verification asks whether the speaker is the person they claim to be. The first needs no prior knowledge of anyone; the other two need enrolled voiceprints, which changes both the engineering and the legal position.
Key takeaways
- Diarization: within one recording, group speech by voice. Output is Speaker 1, Speaker 2. No enrolment needed.
- Identification: compare a voice against many enrolled people and return a name, or "unknown" in open-set systems.
- Verification: compare a voice against one enrolled person and return accept or reject. It is used for authentication.
- All three are built on the same speaker embeddings. What differs is the comparison, the metric and the stored data.
- Identification and verification store voiceprints of known people. Under GDPR that is biometric data used for unique identification, a special category.
Three tasks, three questions#
| Diarization | Identification | Verification | |
|---|---|---|---|
| Question | Who spoke when? | Which enrolled person is this? | Is this the claimed person? |
| Needs enrolment | No | Yes, everyone in the set | Yes, the claimed person |
| Output | Anonymous labels over time | A name, or unknown | Accept or reject, with a score |
| Typical metric | Diarization error rate | Identification accuracy, open-set error rates | Equal error rate, DET curve |
| Typical use | Meeting and call transcripts | Naming known participants, media archives | Phone banking authentication |
| Stored biometric data | Usually none beyond the file | Voiceprints of all enrolled people | Voiceprint of each user |
Diarization#
Diarization works entirely inside one recording. It computes embeddings for short segments and clusters them. The labels are relative: it can say that the voice at 00:12 is the same as the voice at 04:30, but not who that is. Nothing needs to be known about the speakers in advance, and the embeddings can be discarded once the labels are written.
Speaker identification#
Identification compares an unknown voice against a gallery of enrolled speakers and picks the closest. In a closed-set system the speaker is assumed to be one of the enrolled people, so the system always returns a name. In an open-set system it may also answer "none of these", which requires a threshold and makes the problem harder, because every new speaker is a potential false match.
Identification is often layered on top of diarization. The system diarizes a meeting into Speaker 1 to 4, then compares each cluster's average embedding with enrolled colleagues to label them by name. Clusters that match nobody stay anonymous.
Speaker verification#
Verification is a one-to-one test. A user claims an identity, typically through an account number, and says a phrase. The system compares the voice to that account's voiceprint and accepts or rejects based on a score threshold. Verification can be text-dependent (a fixed passphrase) or text-independent (any speech, such as the first seconds of a call).
Two error types are traded against each other: false acceptance (an impostor gets in) and false rejection (the real user is refused). The equal error rate is the point where they are the same, and it is the headline number in benchmarks such as VoxCeleb. In deployment the threshold is set by the cost of each error, and a bank will usually accept more false rejections to reduce false acceptances.
Shared technology#
All three tasks rest on speaker embeddings: vectors computed from speech such that the same voice produces nearby vectors. x-vectors, ECAPA-TDNN and ResNet-based extractors trained on datasets such as VoxCeleb are used across all three. The differences are in how embeddings are compared:
- Diarization clusters many embeddings from one recording.
- Identification ranks one embedding against many stored ones.
- Verification scores one embedding against one stored one, often with PLDA or cosine similarity and a calibrated threshold.
None of them recognises words. That is speech recognition, not voice recognition. They are also independent of language identification, which uses similar embedding models but is trained to separate languages rather than people.
Legal and privacy differences#
Under the GDPR, biometric data means personal data resulting from technical processing of physical or behavioural characteristics that allows or confirms unique identification of a person (Article 4(14)). Processing biometric data for the purpose of uniquely identifying someone is special-category processing under Article 9, which is prohibited unless an exception applies, most commonly explicit consent.
Identification and verification exist precisely to identify people, so they fall squarely into this category. Diarization labels speakers within a recording without linking them to identities, which puts it in a different position, although the audio itself is still personal data. The details depend on purpose and implementation; see voice biometrics and consent and GDPR and voice data. The EU AI Act adds further obligations for remote biometric identification systems.
Common questions#
Can diarization be turned into identification?
Yes, by comparing each diarized cluster with enrolled voiceprints. At that point you are doing identification, with the enrolment, consent and storage requirements that come with it.
Is speaker verification secure against recordings and synthetic voices?
Not by default. Replayed recordings and voice cloning are known attacks, and verification systems need separate spoofing and liveness detection. The ASVspoof challenge series exists to benchmark these countermeasures.
How much audio is needed?
More is better for all three. Embeddings from a second or two of speech are noticeably less reliable than those from ten seconds or more, which is why short backchannels are hard to diarize and why verification systems ask for a phrase rather than a single word.
References#
- Reynolds, D. A., Quatieri, T. F., & Dunn, R. B. (2000). Speaker Verification Using Adapted Gaussian Mixture Models. Digital Signal Processing, 10(1–3).
- Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. (2018). X-vectors: Robust DNN Embeddings for Speaker Recognition. ICASSP.
- Nagrani, A., Chung, J. S., & Zisserman, A. (2017). VoxCeleb: A Large-Scale Speaker Identification Dataset. Interspeech.
- Desplanques, B., Thienpondt, J., & Demuynck, K. (2020). ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. Interspeech.
- Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 4(14) and 9. EUR-Lex.
- Anguera, X. et al. (2012). Speaker Diarization: A Review of Recent Research. IEEE Transactions on Audio, Speech, and Language Processing, 20(2).