Audio event detection identifies non-speech sounds in a recording, such as music, laughter, applause, a dog barking, breaking glass or a siren, and, in its stronger form, when each one starts and stops. It shares front-end features with speech recognition but is a classification problem, not a transcription one. In speech pipelines it is mostly used to find or remove the parts of a recording that are not speech: hold music, ringtones, background television. Detecting those regions reliably also reduces a known ASR failure, where a model writes words over audio that contains none.
Key takeaways
- Audio tagging labels a whole clip; sound event detection labels events with start and end times. The second is much harder.
- Real audio is polyphonic. Speech, music and noise overlap, so models output independent probabilities per class, not a single choice.
- AudioSet, with 632 classes and around two million YouTube clips, is the reference training and evaluation set for general-purpose tagging.
- In speech products, the most valuable events are the ones that should not be transcribed: music, tones and background media.
- Captions for accessibility need sound events too. "[applause]" and "[phone ringing]" are part of a good subtitle file.
Tagging, detection and localisation#
The field uses several task names that differ in what the output must contain:
| Task | Output | Example |
|---|---|---|
| Acoustic scene classification | One label for the environment | "street", "metro station", "office" |
| Audio tagging | Set of labels for a clip | {speech, music, dog} |
| Sound event detection | Labels with onset and offset times | dog 2.4–3.1 s, speech 0.0–8.0 s |
| Sound event localisation | Labels, times and direction of arrival | siren, 5–9 s, from the left |
Voice activity detection is a special case with a single class, speech, and it is the most widely deployed form of audio event detection by a wide margin. Speaker change and overlap detection, used in diarisation, are close relatives.
How detection models work#
Most systems convert audio to a log-mel spectrogram, a time-frequency image, and process it with a convolutional network, a transformer, or a combination. The model outputs a probability per class per frame, typically every 10 to 40 milliseconds. Thresholds turn probabilities into active or inactive, and post-processing such as median filtering removes flicker and merges short gaps.
Hershey et al. (2017) showed that image-classification CNN architectures transfer well to audio tagging when trained on very large data. Kong et al. (2020) released PANNs, pretrained audio neural networks on AudioSet, which became a common starting point for downstream tasks. The Audio Spectrogram Transformer (Gong, Chung and Glass, 2021) applied a vision-transformer design to spectrogram patches. Self-supervised audio encoders now play the same role that pretrained image models play in computer vision: a general representation fine-tuned on a small labelled set.
Datasets and challenges#
- AudioSet (Gemmeke et al., 2017): about two million ten-second YouTube clips labelled with a 632-class ontology. Labels are weak (clip-level, not timed) and noisy, and some clips have since become unavailable, so published results are not always on identical data.
- ESC-50 (Piczak, 2015): 2,000 five-second clips in 50 environmental classes. Small and clean, used for quick comparisons.
- UrbanSound8K (Salamon et al., 2014): 8,732 clips of ten urban sound classes such as drilling, sirens and car horns.
- DCASE: the annual Detection and Classification of Acoustic Scenes and Events challenge, which defines shared tasks, datasets and metrics for scene classification, event detection and localisation.
Evaluation#
Tagging is usually scored with mean average precision (mAP) over classes. Detection needs timing, and Mesaros, Heittola and Virtanen (2016) defined the segment-based and event-based metrics widely used since. Segment-based metrics compare activity in fixed time slices, such as one second. Event-based metrics match each predicted event to a reference event, with a tolerance on onset and offset. The polyphonic sound detection score (PSDS), introduced by Bilen et al. (2020), summarises performance across operating points and was adopted by DCASE.
Scores vary sharply by class. Sirens, alarms and speech are distinctive and detected well; "background noise" or "crowd" are diffuse and hard to label consistently even for humans.
Uses in speech pipelines#
Stopping hallucinations#
Sequence-to-sequence ASR models, trained to always produce text, can generate plausible words over music, silence or noise. On a call, [hold music for 40 seconds] can come back as thank you for watching. Gating the recogniser with a speech detector, or dropping segments a music classifier is confident about, removes much of this. See hallucinations in speech recognition.
Call analytics#
Contact-centre analytics uses event detection to measure hold time, detect ringback and voicemail greetings, and flag dual-tone multi-frequency (DTMF) keypad tones. Separating an answering machine from a live person at the start of an outbound call is a classic application.
Captions and media#
Subtitles for deaf and hard-of-hearing viewers include non-speech information such as [laughter], [door slams] or [music]. Broadcast workflows also use music detection for rights reporting. See captions and subtitles.
Safety and monitoring#
Outside speech products, event detection is used for glass-break and smoke-alarm detection in home security, machine-fault monitoring from equipment sound, and wildlife surveys. These rarely need a transcript at all.
Limits#
- Label noise. Large datasets are weakly and inconsistently labelled, so model confidence is less meaningful than it looks.
- Domain shift. A model trained on YouTube audio meets an 8 kHz phone line, a car cabin or a factory floor. Retest in the target environment.
- Overlap. Quiet events under loud ones, such as a phone ring under speech, are often missed.
- Privacy. Continuous monitoring of sound in homes or workplaces raises the same questions as recording speech, even when no words are transcribed.
How Reson8 approaches this#
Reson8 does not offer a general sound-event classifier. Its concern is the speech parts of a recording: producing accurate transcripts from real-world audio, including noisy and telephony input, which downstream event detectors and analytics can then be aligned with using word timestamps.
Common questions#
Is audio event detection part of speech recognition?
It is a separate task, though often run alongside it. Speech recognition decides which words were said; event detection decides what kinds of sound are present and when.
Can one model do both?
Some large audio models produce transcripts and sound tags together. Most production pipelines still use a dedicated detector because it is small, fast and easy to tune.
How much audio do I need to add a new class?
Fine-tuning a pretrained audio encoder can work with a few hundred labelled examples for a distinctive sound. Diffuse or variable sounds need far more, plus plenty of negatives from your environment.
References#
- Gemmeke, J. F. et al. (2017). Audio Set: An ontology and human-labeled dataset for audio events. ICASSP.
- Hershey, S. et al. (2017). CNN architectures for large-scale audio classification. ICASSP.
- Kong, Q. et al. (2020). PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28.
- Gong, Y., Chung, Y.-A., & Glass, J. (2021). AST: Audio Spectrogram Transformer. Interspeech.
- Mesaros, A., Heittola, T., & Virtanen, T. (2016). Metrics for Polyphonic Sound Event Detection. Applied Sciences, 6(6).
- Bilen, Ç. et al. (2020). A Framework for the Robust Evaluation of Sound Event Detection. ICASSP.
- Piczak, K. J. (2015). ESC: Dataset for Environmental Sound Classification. ACM Multimedia.
- DCASE Community. dcase.community.