Audio intelligence is the umbrella term for extracting useful structure from recordings beyond the words themselves: who spoke, what the conversation was about, what was agreed, how people sounded, and what else could be heard. Most systems build it on top of a transcript, so recognition accuracy on names, numbers and domain terms sets a ceiling on everything downstream. A smaller set of capabilities, such as audio event detection and prosody, works on the signal directly.
Key takeaways
- Audio intelligence covers diarization, summarisation, topic and intent detection, entity extraction, sentiment, redaction and audio event detection.
- Most capabilities are text-derived and run on the transcript. A few, such as events, speaker identity and tone, need the audio.
- In a cascaded pipeline, recognition errors propagate. A misheard drug name or account number becomes a wrong summary or a wrong CRM field.
- Each capability has its own metric. Word error rate on the transcript does not tell you how good the summary is.
- Some capabilities carry legal weight: emotion recognition from voice is restricted by the EU AI Act, and speaker identification is biometric processing under GDPR.
What it includes#
The term is used by vendors and analysts for a loosely defined set of features. They split naturally by what they operate on.
| Capability | Operates on | Typical output |
|---|---|---|
| Speaker diarization | Audio | Speaker labels with time ranges |
| Language identification | Audio | Language per segment |
| Summarisation | Transcript | Notes, action items, decisions |
| Topic and intent detection | Transcript | Labels such as "cancellation" or "billing query" |
| Entity extraction | Transcript | Names, products, amounts, dates |
| Sentiment | Transcript, sometimes audio | Positive, negative or neutral per turn |
| PII redaction | Transcript and timestamps | Masked text and muted audio |
| Keyword spotting | Audio or transcript | Hits for specific phrases |
| Audio event detection | Audio | Events such as music, alarms, laughter |
How it is built#
Cascaded pipelines#
The dominant design transcribes first and analyses the text afterwards. A recogniser produces words with timestamps and speaker labels, and text models, increasingly general large language models, run summarisation, classification and extraction over the result. A separate acoustic branch handles anything that is not in the words.
Cascades are modular. Each stage can be swapped, evaluated and audited on its own, the transcript is a human-readable intermediate that can be checked, and text models are cheap to run compared with audio models. The cost is that information the transcript does not capture, such as tone, hesitation or overlapping speech, is lost to the later stages.
Audio-language models#
A newer approach feeds audio directly into a model that can answer questions about it, summarise it or classify it. These models can in principle use tone and timing as well as words. They are harder to evaluate, since there is no intermediate transcript to inspect, and when they are asked for a summary they can produce fluent text that the audio does not support, much as a recogniser can hallucinate words. Many production systems still produce a transcript anyway, because users and auditors want to see it.
Error propagation#
In a cascade, every downstream feature inherits the recogniser's mistakes. The words that matter most to analytics are usually the words general models find hardest: product names, people's names, medication, account numbers and amounts.
Consider a support call in which the customer says my order INV762 hasn't arrived and the recogniser returns my order in the seven six two hasn't arrived. The summary will report a delivery complaint with no order number, the CRM lookup will fail, and a topic model may still get the category right. The error is one entity, but it breaks the only field anyone needed.
Evaluating audio intelligence#
Each capability needs its own metric, and a strong transcript does not guarantee strong analytics.
- Diarization is scored with diarization error rate, which combines missed speech, false alarms and speaker confusion.
- Summaries are often scored with ROUGE, which measures word overlap with reference summaries. It rewards overlap rather than correctness, so human review or targeted fact checks remain necessary.
- Entities, topics and intents are classification or extraction tasks, scored with precision, recall and F1 per label.
- Sentiment needs labelled data from your own domain, since what counts as negative differs between a complaints line and a sales call.
Academic benchmarks such as SUPERB and SLUE evaluate spoken language understanding tasks across shared datasets, and AudioSet provides a large labelled collection for audio event classification. None of them replaces a test set built from your own recordings; see building your own ASR test set.
Legal limits#
Some audio intelligence features are regulated in their own right. Under the EU AI Act, inferring emotions from voice is prohibited in workplaces and education, except for medical or safety reasons, and is high-risk elsewhere. Monitoring the emotional state of call centre agents from their voices falls on the prohibited side. Linking voices to named people across recordings is biometric processing under GDPR; see voice biometrics and consent. Summaries and extracted entities of calls remain personal data and need the same retention and access controls as the recordings.
How Reson8 approaches this#
Reson8 provides the recognition layer that text-derived audio intelligence depends on: real-time streaming, batch transcription and the Turns endpoint for voice agents. Domain adaptation lets customers supply domain text through the API, up to 1M tokens of context, applied at runtime without retraining. Patterns biases recognition toward expected identifier formats; on order numbers it reduced WER from 8.0% to 1.5%. Both aim at the entities that downstream analytics needs most. Details are in the documentation.
Common questions#
Is audio intelligence the same as speech analytics?
Largely. "Speech analytics" is the older term, especially in contact centres. "Audio intelligence" is broader and includes non-speech sounds.
Do I need a special model for summaries, or can I use a general LLM?
General language models summarise transcripts well, provided the transcript is accurate and speaker-labelled. The quality of the input usually matters more than the choice of model.
Can sentiment be detected from text alone?
Yes, and most systems do exactly that. Text misses sarcasm and tone; acoustic features add them but bring the legal questions around emotion recognition.
Should analysis run in real time or after the call?
Agent assist and live alerts need streaming. Summaries, quality scoring and trend analysis usually work better on the complete call, after batch transcription.
References#
- Yang, S. et al. (2021). SUPERB: Speech processing Universal PERformance Benchmark. Interspeech.
- Shon, S. et al. (2022). SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech. ICASSP.
- Gemmeke, J. F. et al. (2017). Audio Set: An ontology and human-labeled dataset for audio events. ICASSP.
- Bredin, H. et al. (2020). pyannote.audio: neural building blocks for speaker diarization. ICASSP.
- Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, ACL Workshop.
- Directive 2014/65/EU on markets in financial instruments (MiFID II), Article 16(7). EUR-Lex.
- Regulation (EU) 2024/1689 (Artificial Intelligence Act). EUR-Lex.