Sentiment analysis on speech estimates whether a speaker is positive, negative or neutral about something, from what they say, how they say it, or both. Text-based methods run on a transcript and are mature and cheap; acoustic methods analyse pitch, energy and timing and are less reliable outside the lab. Both capture a narrow signal. Sarcasm, politeness conventions, culture and context routinely defeat them, so sentiment scores are best used to find calls worth reviewing, not to judge individuals.
Key takeaways
- Most production speech sentiment is text sentiment on a transcript. Its accuracy depends on the transcript, especially negations.
- Acoustic emotion recognition reads voice properties such as pitch and loudness. Benchmarks are mostly acted or elicited speech, and performance drops on real calls.
- Sentiment is not emotion, and neither is intent. A calm customer can be about to cancel.
- Scores are most useful aggregated: trends across thousands of calls, or ranking calls for human review.
- In the EU, inferring emotions of people at work or in education is prohibited under the AI Act, with narrow medical and safety exceptions.
What sentiment analysis measures#
The term covers several related tasks that are often blurred in product descriptions:
| Task | Output | Main signal |
|---|---|---|
| Sentiment (polarity) | Positive / neutral / negative, or a score from −1 to 1 | Words |
| Aspect-based sentiment | Polarity toward a named topic ("billing: negative") | Words |
| Speech emotion recognition | Categories (angry, happy, sad, neutral) or dimensions (arousal, valence) | Voice |
| Escalation or churn risk | Probability of an outcome | Words, voice and metadata |
Emotion research often uses two dimensions instead of categories: valence (pleasant to unpleasant) and arousal (calm to excited). Anger and joy are both high-arousal; sadness and contentment are both low. Acoustic features predict arousal much better than valence, because loudness and pitch range track excitement fairly directly, while whether that excitement is pleasant is mostly carried by words.
Text-based sentiment on transcripts#
The simplest pipeline transcribes the call, then classifies each turn or sentence. Lexicon methods count positive and negative words. Trained classifiers, and now general-purpose language models, handle context better: "not bad at all" is positive, "I'd love to, but" is not.
Text sentiment inherits every weakness of the transcript. Negation is the critical case. A single missed word reverses polarity:
- I'm not happy with the fix heard as I'm now happy with the fix
- that didn't help heard as that did help
Each of these counts as one word error in WER and flips the label. Speaker attribution matters too: an agent reading out a complaint policy ("I understand this is frustrating") should not be scored as a frustrated customer. Diarisation or separate recording channels are a prerequisite for per-speaker sentiment.
Acoustic emotion recognition#
Speech emotion recognition works on the signal. Classic systems extract hand-designed features such as pitch contour, energy, speaking rate, jitter and shimmer, and spectral shape. The Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) standardised a compact feature set for this purpose. Newer systems fine-tune self-supervised speech encoders on labelled emotional speech.
The main limitation is data. Widely used corpora such as IEMOCAP consist of actors performing scripted or improvised scenes in a studio. Acted emotion is exaggerated and clearly labelled; real customers on a noisy 8 kHz phone line are neither. Annotators also disagree often on real emotional speech, which caps how accurate any model trained on their labels can be. Accuracy reported on acted benchmarks should not be read as accuracy on production calls.
Voice properties also vary for reasons that have nothing to do with emotion: individual voice, age, native language, a cold, the microphone, and the codec. Someone who speaks loudly and quickly by habit can read as agitated.
Combining words and voice#
Multimodal models combine text and acoustic features, and datasets such as CMU-MOSEI and MELD were built to study this. The acoustic channel adds most where words are ambiguous: "fine" said flatly after a long wait is not the same as "fine" said brightly. Research systems usually find that text carries most of the signal for polarity, with audio contributing a smaller improvement, mostly on arousal.
A cheaper and often more useful proxy is conversational behaviour, which the transcript already contains: long silences, frequent interruptions, raised speaking rate, repeated requests for a supervisor, or the same question asked three times. These are easy to explain to a supervisor, which matters when a score leads to action.
Limits and risks#
- Sarcasm and irony. "Brilliant, another hour on hold" has positive words and negative meaning.
- Culture and language. Directness, politeness and emotional expression differ across languages and regions. A model calibrated on American English support calls may misread Dutch or German directness as hostility.
- Domain. In a clinical consultation, words such as "positive", "negative" and "aggressive" have technical meanings.
- Individual judgement. Scores are noisy per call and more stable in aggregate. Using them to rate individual employees invites unfair outcomes.
Text sentiment about a product ("the invoice is wrong and I'm annoyed") is generally treated differently from inferring a person's emotional state from biometric signals such as their voice, but the boundary deserves legal review before deployment. Voice features can also qualify as biometric data under GDPR; see voice biometrics and consent.
How Reson8 approaches this#
Reson8 does not offer sentiment or emotion analysis. Its role is the transcript that text-based sentiment runs on, in nine European languages, which matters for teams analysing calls outside English. Accurate negations, product names (via text-only domain adaptation) and clean speaker turns are what a downstream classifier needs.
Common questions#
Is voice-based emotion detection accurate?
On acted benchmark data, reasonably. On real calls, much less so, and results vary by speaker, language and audio channel. Treat it as a weak signal that helps rank calls, not as a measurement.
Do I need acoustic analysis, or is the transcript enough?
For most contact-centre uses, transcript-based sentiment plus conversational behaviour (silence, interruptions, repeats) captures most of the useful signal and is easier to explain and audit.
Can I use a general LLM for sentiment?
Yes, and it handles negation and context better than lexicon methods. Give it speaker-labelled turns, ask for a label per turn with a short justification, and check it against a human-labelled sample from your own calls. See using transcripts with LLMs.
Should sentiment be computed in real time?
Only if someone will act on it during the call, such as a supervisor alert. Otherwise post-call analysis on a batch transcript is cheaper and more accurate.
References#
- Busso, C. et al. (2008). IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4).
- Eyben, F. et al. (2016). The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing. IEEE Transactions on Affective Computing, 7(2).
- Schuller, B., Steidl, S., & Batliner, A. (2009). The INTERSPEECH 2009 Emotion Challenge. Interspeech.
- Zadeh, A. et al. (2018). Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. ACL.
- Poria, S. et al. (2019). MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. ACL.
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 5. EUR-Lex.