Conversation summarisation turns a spoken exchange, such as a support call, a meeting or a consultation, into a short written record of what was discussed, decided and left open. In almost every production system it is a two-stage process: speech recognition produces a transcript with speaker labels, and a language model condenses that transcript. The quality of the summary is bounded by the quality of the transcript, and the hardest problem is not fluency but faithfulness: making sure the summary only says things that were actually said.
Key takeaways
- Summaries are built from transcripts. A misheard name, amount or negation in the transcript becomes a confident error in the summary.
- Speaker attribution matters as much as words. "The customer agreed" and "the agent agreed" are different outcomes.
- Modern systems are abstractive: a large language model writes new sentences. That makes them readable, and able to invent content.
- Structured output (fixed fields such as reason, actions, owner, due date) is easier to check and to feed into other systems than free prose.
- Automatic metrics such as ROUGE measure word overlap with a reference summary, not correctness. Human review on a sample remains the standard check.
How conversation summarisation works#
A conversation summary pipeline has four stages. Audio is transcribed by an ASR model, speakers are separated by diarisation or by recording each party on its own channel, the transcript is cleaned and formatted, and a language model produces the summary.
Before large language models, summarisation research split into two families. Extractive methods pick the most informative sentences from the source and stitch them together. Abstractive methods write new text. Extractive summaries of conversations read poorly, because spoken turns are fragmentary, full of false starts and depend on the previous turn for meaning. A sentence such as "yeah, that one, the second" is useless out of context. Abstractive summarisation with an instruction-tuned LLM is now the default, and extractive techniques survive mainly as a way to ground or cite the abstractive output.
Why conversations are harder than documents#
Summarising a news article and summarising a call are different tasks. Written text is edited; speech is not. A conversation summary has to cope with several properties that documents rarely have:
- Information is spread across turns. A question in minute two may only be answered in minute fourteen, after a hold.
- Decisions are implicit. People rarely say "we have decided". Agreement is a "fine, let's do that" after a proposal.
- Corrections and reversals. "Thursday. No, sorry, Friday." The summary must carry the final value, not the first.
- Two or more perspectives. The customer's complaint and the agent's explanation are both content, and they may contradict each other.
- Disfluency and noise. Fillers, overlaps and noisy audio add material that should not appear in the output.
Research corpora reflect this. The AMI and ICSI meeting corpora provide recorded meetings with human-written summaries, and QMSum adds query-based summarisation, where the model answers a specific question such as "what did the group decide about the budget?" rather than summarising everything.
Transcript quality sets the ceiling#
A language model cannot recover information the transcript lost. Some ASR errors are harmless to a summary; a dropped "um" or a misheard filler word changes nothing. Others propagate directly:
| Transcript error | Effect on the summary |
|---|---|
| I can't make the fifteenth heard as I can make the fifteenth | The summary records a confirmed appointment that was declined. |
| order INV7625X heard as order INB seven six two five X | The action item references an order that does not exist. |
| Agent and customer turns merged under one speaker | Promises made by the customer are attributed to the company. |
| A hallucinated sentence during hold music | The summary includes a topic nobody discussed. |
This is why aggregate word error rate is a weak predictor of summary quality. What matters is accuracy on the words that carry outcomes: names, numbers, dates, negations and identifiers. Entity and keyword accuracy and semantic WER track these better. For phone calls, recording each party on a separate channel removes the diarisation problem entirely and is usually worth the setup effort.
Output formats#
The right format depends on who reads the summary and what happens next.
Free-text notes#
A paragraph or a few bullet points. Easy to read, hard to search or validate. Suitable for meeting recaps sent to participants.
Structured records#
A fixed schema, such as contact reason, resolution status, follow-up actions, owner and due date. Contact centres use this to replace the notes agents type after each call, often called after-call work. Structured output can be validated automatically: a due date must parse as a date, an order number must match a known format, a status must be one of a fixed set of values.
Domain templates#
Clinical notes follow formats such as SOAP (subjective, objective, assessment, plan). Ambient clinical documentation is conversation summarisation with a strict template, a specialist vocabulary and a clinician who signs off the result. Legal and financial settings often need a verbatim transcript kept alongside any summary, because the summary is not the record.
Long conversations#
A one-hour meeting produces something in the region of 8,000 to 10,000 spoken words. Current language models can take that in a single prompt, but long inputs bring their own problems. Liu et al. (2023) showed that models use information at the start and end of a long context more reliably than information in the middle, which for a meeting means the decisions made halfway through are the ones most at risk.
Two common strategies:
- Map-reduce. Split the transcript into chunks, summarise each, then summarise the summaries. Robust and parallel, but decisions that span chunk boundaries can be lost or duplicated.
- Extract then write. First ask the model to list every decision, number and action item with its timestamp; then write the summary from that list. The intermediate list is short, checkable and reusable.
For recordings of several hours, see long-form transcription for the transcription side of the problem.
Evaluating summaries#
ROUGE (Lin, 2004) measures n-gram overlap between a generated summary and a human reference. It is cheap and reproducible, and it correlates poorly with what users care about: a summary can score well on ROUGE while reversing the outcome of the call. Embedding metrics such as BERTScore capture paraphrase better but share the same blind spot.
Faithfulness needs its own check. Maynez et al. (2020) found that abstractive summarisation models frequently produce content not supported by the source. For conversations, practical evaluation usually combines:
- a human review of a random sample, scored for missing facts, unsupported facts and wrong attribution;
- automatic checks on structured fields (valid dates, known identifiers, allowed status values);
- LLM-as-judge comparisons against the transcript, calibrated against the human sample before they are trusted.
How Reson8 approaches this#
Reson8 does not provide summarisation. It provides the transcript that summarisation depends on. Two features target the words that summaries most often get wrong: Patterns, which biases recognition toward expected identifier formats (order numbers went from 8.0% to 1.5% WER in Reson8's tests), and text-only domain adaptation for product names and specialist terms. Batch transcription suits post-call summaries; processing runs on EU GPUs with zero audio retention.
Common questions#
Can a model summarise audio directly, without a transcript?
Some multimodal models accept audio input. Most production systems still produce a transcript first, because it can be stored, searched, redacted and used as evidence when a summary is disputed.
How accurate does the transcript need to be?
Accurate on outcome-bearing words: names, numbers, dates, negations and identifiers. A transcript with a few percent WER concentrated on fillers produces good summaries; one with the same WER concentrated on amounts does not.
Should the summary replace the transcript?
Not in regulated settings. Keep the transcript, or at least the ability to regenerate it, so any statement in the summary can be traced to what was said.
Real-time or after the call?
Most summaries are generated after the call from a batch transcript, which is cheaper and usually more accurate. Rolling summaries during a call are useful for agent handover or live assistance.
References#
- Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. ACL Workshop on Text Summarization Branches Out.
- Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020). On Faithfulness and Factuality in Abstractive Summarization. ACL.
- Carletta, J. et al. (2005). The AMI Meeting Corpus: A Pre-announcement. Machine Learning for Multimodal Interaction (MLMI).
- Janin, A. et al. (2003). The ICSI Meeting Corpus. ICASSP.
- Zhong, M. et al. (2021). QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization. NAACL.
- Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
- Zhang, T. et al. (2020). BERTScore: Evaluating Text Generation with BERT. ICLR.