Speaker diarization answers the question "who spoke when?". It splits a recording into segments and gives each segment an anonymous speaker label, such as Speaker 1 and Speaker 2, so that segments with the same label come from the same voice. It does not know who the speakers are. Combined with a transcript, it turns a wall of text into a readable conversation.
Key takeaways
- Diarization labels speakers relatively, not by identity. Naming them is speaker identification, a different task.
- The classic pipeline is VAD, segmentation, speaker embeddings and clustering. End-to-end neural models now handle overlap far better.
- Overlapping speech, short turns and an unknown number of speakers are the hard cases. Two-person phone calls on separate channels need no diarization at all.
- Diarization error rate (DER) adds missed speech, false alarms and speaker confusion over total speech time. Scoring settings, especially the collar and whether overlap counts, change the number a lot.
- Word-level speaker labels depend on accurate word timestamps. A good diarization with poor timing still puts words in the wrong mouth.
What diarization produces#
The raw output is a list of time ranges with speaker labels. The standard file format is RTTM (Rich Transcription Time Marked), from the NIST Rich Transcription evaluations: one line per segment with file, start time, duration and speaker label. Merged with a timestamped transcript, it becomes:
[00:00.4 – 00:03.1] Speaker 1: Good morning, how can I help?
[00:03.3 – 00:07.9] Speaker 2: I'm calling about an invoice from March.
[00:08.0 – 00:08.6] Speaker 1: Of course.
Speaker labels are consistent within a recording but carry no meaning across recordings. Speaker 1 in one call and Speaker 1 in the next are unrelated.
How diarization works#
Voice activity detection#
A VAD removes silence and non-speech. Any speech it misses is counted as missed speech in DER, and any noise it accepts will be assigned to some speaker.
Segmentation#
Speech is cut into short pieces that are each assumed to contain one speaker, either at fixed windows of a second or two or at detected speaker change points.
Speaker embeddings#
Each piece is turned into a vector that captures the voice rather than the words. Early systems used i-vectors; x-vectors (Snyder et al., 2018) replaced them with a neural network trained to classify thousands of speakers, keeping an internal layer as the embedding. ECAPA-TDNN and ResNet-based extractors are common today.
Clustering#
Embeddings are grouped so that each cluster is one speaker. Agglomerative hierarchical clustering and spectral clustering are standard. If the number of speakers is known, pass it in: estimating it is one of the main sources of error.
End-to-end and hybrid models#
The modular pipeline assumes one speaker per segment, so it handles overlap badly. End-to-end neural diarization (EEND, Fujita et al., 2019) predicts, frame by frame, which of several speakers is active, and can mark more than one at once. Current open-source systems such as pyannote.audio combine the two: a neural model handles segmentation and overlap on short windows, and embedding clustering links speakers across the whole recording.
from pyannote.audio import Pipeline
pipeline = Pipeline.from_pretrained(
"pyannote/speaker-diarization-3.1", use_auth_token="HF_TOKEN")
diarization = pipeline("call.wav", num_speakers=2)
for turn, _, speaker in diarization.itertracks(yield_label=True):
print(f"{turn.start:.1f}-{turn.end:.1f} {speaker}")
Measuring diarization: DER#
Speaker labels are arbitrary, so scoring first finds the best one-to-one mapping between hypothesis and reference speakers, then measures time. Speaker confusion is speech assigned to the wrong speaker; the other two terms come mostly from the VAD.
Scoring conventions vary. NIST evaluations traditionally ignored a 0.25 s collar around each reference boundary, because human labels are imprecise there, and some published results also exclude overlapped speech. Both choices lower DER substantially. The DIHARD challenges were set up partly to push evaluation towards harder audio with no collar and overlap included. As with WER, a DER is only comparable with another computed on the same data with the same settings.
Where diarization struggles#
| Condition | Why it is hard | Mitigation |
|---|---|---|
| Overlapping speech | Two voices in one segment; embeddings blur | Overlap-aware neural segmentation |
| Short turns and backchannels | Too little audio for a reliable embedding | Smaller windows, resegmentation passes |
| Unknown speaker count | Clustering must guess when to stop merging | Pass the count if known; set min and max |
| Similar voices | Embeddings sit close together | Longer context; channel information if available |
| Far-field and reverberant rooms | Room acoustics dominate the embedding | Close microphones, multi-channel beamforming |
Meeting audio combines most of these, which is why meeting benchmarks such as AMI remain difficult for both transcription and diarization. See transcribing noisy audio.
Where diarization is used#
- Meetings and interviews: readable minutes and summaries with who said what.
- Contact centres: separating agent and customer for speech analytics, talk-time ratios and compliance checks.
- Clinical consultations: distinguishing clinician from patient in ambient documentation.
- Legal and media: hearings, depositions and broadcast archives where attribution is part of the record.
Common questions#
Does diarization tell me the speakers' names?
No. It produces anonymous labels. Assigning names requires enrolled voice samples (speaker identification) or context, such as knowing the agent always speaks first. See diarization vs identification vs verification.
Is a voice embedding personal data?
It can be. Under GDPR, biometric data processed to uniquely identify a person is special-category data. Diarization that only labels speakers within one file is usually treated differently from building a persistent voiceprint, but check with your data protection officer. See voice biometrics and consent.
Can diarization run in real time?
Yes, with trade-offs. Online diarization must assign a label before it has heard the whole recording, so early labels are less reliable and speakers who first speak late can be merged with earlier ones. Offline diarization over the full file is more accurate.
What is a good DER?
It depends heavily on the audio and scoring settings. Clean two-speaker calls can be low single digits; overlapping meeting audio scored without a collar is much higher. Compare systems only on your own recordings with one fixed scoring setup.
References#
- Anguera, X., Bozonnet, S., Evans, N., Fredouille, C., Friedland, G., & Vinyals, O. (2012). Speaker Diarization: A Review of Recent Research. IEEE Transactions on Audio, Speech, and Language Processing, 20(2).
- Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. (2018). X-vectors: Robust DNN Embeddings for Speaker Recognition. ICASSP.
- Fujita, Y., Kanda, N., Horiguchi, S., Nagamatsu, K., & Watanabe, S. (2019). End-to-End Neural Speaker Diarization with Permutation-Free Objectives. Interspeech.
- Park, T. J., Kanda, N., Dimitriadis, D., Han, K. J., Watanabe, S., & Narayanan, S. (2022). A Review of Speaker Diarization: Recent Advances with Deep Learning. Computer Speech & Language, 72.
- Ryant, N. et al. (2019). The Second DIHARD Diarization Challenge: Dataset, task, and baselines. Interspeech.
- Bredin, H. et al. (2020). pyannote.audio: neural building blocks for speaker diarization. ICASSP. GitHub.