Voice activity detection (VAD) decides, frame by frame, whether an audio signal contains human speech. It is the smallest detector in a speech pipeline and one of the most consequential: it decides what audio reaches the recogniser, when a speaker is considered to have started, and when silence begins. VAD answers "is someone speaking right now?" and nothing more.
Key takeaways
- VAD is a binary classifier on short frames, usually 10 to 32 ms. Its raw output is noisy and always smoothed before use.
- Energy thresholds work in quiet rooms and fail in noise. Modern VADs are small neural networks that distinguish speech from music, typing and traffic.
- Missed speech is usually worse than false alarms: clipped words are gone, while extra noise sent to the recogniser mostly costs compute.
- VAD does not know whether a speaker has finished a thought. That is the job of endpointing and turn detection.
- Feeding long silences to a recogniser is a known trigger for hallucinated text, which is one reason VAD sits in front of many batch pipelines.
What VAD does#
A VAD splits incoming audio into fixed frames and assigns each a speech probability or a yes/no label. Those labels are then smoothed into segments: stretches of speech with a start and end time. Everything downstream consumes the segments, not the raw frames.
Typical uses:
- Gating a recogniser. Only send speech to the ASR model, saving compute and bandwidth. Telephone codecs have done this for decades: ITU-T G.729 Annex B pairs a VAD with discontinuous transmission so that silence is not transmitted as full frames.
- Chunking long audio. Cut hours of recording at natural pauses so each chunk fits the model's context. See long-form transcription.
- Detecting onset in live systems. A voice agent uses speech onset to notice that the caller has started talking, including while the agent is still speaking. See turn-taking and barge-in.
- Feeding endpointers. The simplest endpointer is a VAD plus a silence timer.
- Preprocessing for diarization. Speaker diarization starts by finding where speech is.
How VAD works#
Energy and spectral rules#
The oldest VADs compare frame energy with an estimate of the background level, often adding zero-crossing rate or spectral features. They are cheap and work on clean, close-talk audio. They fail when the noise is itself loud or speech-like: a television, a café, a car at motorway speed. Statistical-model VADs such as Sohn, Kim and Sung's likelihood-ratio test improved on this by modelling speech and noise spectra separately.
Classical classifiers#
The VAD in the WebRTC project, widely used through the py-webrtcvad wrapper, uses Gaussian mixture models over sub-band energies. It accepts 10, 20 or 30 ms frames of 16-bit mono audio at 8, 16, 32 or 48 kHz, and has an aggressiveness setting from 0 to 3. It is fast and dependable on telephone audio, but treats many non-speech sounds as speech.
Neural VADs#
Current practice is a small neural network, often under a few megabytes, that outputs a speech probability per frame. Silero VAD is a common open-source example. pyannote.audio includes a segmentation model that performs VAD alongside overlap detection. Neural VADs are far better at rejecting music, keyboard noise and background chatter, at the cost of more compute per frame, which is still negligible next to the recogniser.
Smoothing: from frames to segments#
Raw frame decisions flicker. Speech contains natural dips (stop consonants, short pauses between words), and noise produces isolated false positives. Every usable VAD applies post-processing, and these parameters matter as much as the classifier:
| Parameter | What it does | Set too low | Set too high |
|---|---|---|---|
| Threshold | Probability above which a frame counts as speech | Noise becomes speech | Quiet speech is missed |
| Minimum speech duration | Discards very short speech runs | Clicks and coughs pass | Short replies ("ja", "no") are dropped |
| Hangover | Keeps speech on after the last speech frame | Word endings are clipped | Segments absorb trailing noise |
| Pre-roll | Prepends audio from before the onset | First phoneme is lost | Extra audio, small cost |
| Minimum silence to split | Gap needed to end a segment | Sentences split mid-phrase | Separate utterances merge |
Onset detection needs a little delay to be reliable: a detector that fires on the first speech-like frame will also fire on every door slam. Most systems require a few consecutive speech frames before declaring onset, then use pre-roll to recover the audio they waited for.
import webrtcvad
vad = webrtcvad.Vad(2) # aggressiveness 0-3
sample_rate = 16000
frame_ms = 30
frame_bytes = int(sample_rate * frame_ms / 1000) * 2 # 16-bit mono
def speech_flags(pcm: bytes):
for i in range(0, len(pcm) - frame_bytes + 1, frame_bytes):
yield vad.is_speech(pcm[i:i + frame_bytes], sample_rate)
Failure modes#
- Clipped onsets and endings. Soft initial consonants and trailing sounds fall below threshold. The recogniser then sees ix instead of six.
- Noise accepted as speech. Music, TV and other people in the room open segments. The recogniser may transcribe them, or produce plausible-looking text from nothing.
- Quiet or distant speakers missed. Far-field audio and soft voices sit close to the noise floor. See transcribing noisy audio.
- Codec artefacts. Telephone codecs with their own silence suppression insert comfort noise, which some VADs treat as speech. See telephony transcription.
- Echo. In a voice agent, the agent's own voice leaking back through the caller's microphone looks exactly like speech unless echo cancellation removes it first.
Evaluating a VAD#
VAD is scored at frame level against human speech/non-speech labels. The two error types are measured separately: missed speech (speech labelled as silence) and false alarm (non-speech labelled as speech). Plotting both across thresholds gives a detection error trade-off curve. For most transcription work, pick the operating point by missed-speech rate first, because a missed word cannot be recovered later in the pipeline, then check that false alarms do not produce hallucinations or inflate cost.
The end-to-end test is simpler: run the full pipeline with and without the VAD, or with two settings, and compare word error rate on the same audio. Deletions concentrated at segment edges point to hangover or pre-roll being too short.
Common questions#
Is VAD the same as endpoint detection?
No. VAD reports whether speech is present in the current frame. Endpoint detection decides that an utterance is complete, which often requires ignoring pauses the VAD correctly labels as silence. See VAD vs endpointing vs turn detection.
Do modern ASR models still need a VAD?
Many do not need one to transcribe, since they learn to output nothing for silence. VAD is still used to save compute, to chunk long audio, to detect speech onset in real time, and to reduce hallucinations on silent stretches.
Can a VAD tell speakers apart?
No. It detects the presence of speech, not whose speech it is. Separating speakers is diarization.
What frame size should I use?
Whatever the detector was designed for. WebRTC VAD accepts 10, 20 or 30 ms; neural VADs usually expect a fixed chunk size in the region of 30 ms. Smaller frames react faster but are noisier, which smoothing then has to absorb.
References#
- ITU-T Recommendation G.729 Annex B (1996). A silence compression scheme for G.729 optimized for terminals conforming to Recommendation V.70. ITU.
- Sohn, J., Kim, N. S., & Sung, W. (1999). A statistical model-based voice activity detection. IEEE Signal Processing Letters, 6(1).
- Ramírez, J., Segura, J. C., Benítez, C., de la Torre, Á., & Rubio, A. (2004). Efficient voice activity detection algorithms using long-term speech information. Speech Communication, 42(3–4).
- Silero VAD. GitHub.
- py-webrtcvad: Python interface to the WebRTC voice activity detector. GitHub.
- Bredin, H. et al. (2020). pyannote.audio: neural building blocks for speaker diarization. ICASSP.