Large language models do most of the work that happens after a transcript is made: summaries, question answering, classification, extraction of fields, and the reasoning inside a voice agent. The model only sees text, so the transcript is its entire view of the conversation. Getting good results depends on three things: an accurate transcript on the words that matter, a format that keeps speakers and timing, and a design that treats the LLM's output as something to check rather than a record of fact.
Key takeaways
- An LLM cannot recover what the transcript lost. Errors on names, numbers and negations flow straight into its answers.
- Keep speaker labels and timestamps in the prompt. They let the model attribute statements and let people verify them.
- Ask for structured output with evidence (a timestamp or quote per field) wherever the result feeds another system.
- Long recordings fit in modern context windows, but models use the middle of a long input less reliably. Extract first, then reason.
- Transcripts are untrusted input. Anything a caller says can end up in the prompt, including instructions.
The transcript-to-LLM pipeline#
Most systems follow the same shape. Audio is transcribed, with speaker labels and word timestamps. The transcript is cleaned: personal data redacted where required, and formatting applied. It is then placed into a prompt with instructions and, often, an output schema. The response is validated before it is stored or acted on.
In a voice agent the same pipeline runs turn by turn, in real time, and the LLM's output goes to text-to-speech instead of a database. The constraints shift toward latency, but the dependence on transcript quality is the same.
Formatting transcripts for a model#
A raw stream of words without punctuation, speakers or timing is the worst input. A useful default is one line per speaker turn:
[00:03:12] Agent: Can you confirm the order number?
[00:03:15] Customer: Yes, it's INV7625X.
[00:03:19] Agent: Thanks. I can see it was dispatched on the ninth.
[00:03:24] Customer: It hasn't arrived, and I'm not paying twice.
Some practical rules:
- Use role names, not "Speaker 1". If you know which channel is the agent, say so. The model otherwise has to guess roles from content.
- Keep timestamps at turn level. Word-level times add many tokens and rarely help the model; keep them in your own store for linking back.
- Use formatted text. Punctuation, casing and inverse text normalisation ("€20" rather than "twenty euros") make the transcript read like the text LLMs are trained on.
- Remove non-speech hallucinations first. Text generated over hold music or silence becomes content the model will summarise. See hallucinations.
How transcription errors propagate#
An LLM is fluent enough to hide transcript errors. Given I can make the fifteenth where the caller said I can't make the fifteenth, it will confidently report a confirmed booking. Given met for min, it may guess "metformin" correctly, or guess wrong with equal confidence.
Language models do repair some errors from context, which is one reason semantic WER is a better predictor of downstream quality than plain WER. What they cannot repair are errors that produce another plausible reading: a wrong digit in an amount, a similar-sounding name, a dropped negation. Those are exactly the errors worth measuring with entity accuracy and fixing upstream with context biasing or identifier formats.
Some teams pass confidence scores or alternative hypotheses to the model so it can flag uncertain words. This helps when the model is told what to do with them, for example "if a number has low confidence, ask the caller to confirm it", and adds noise otherwise.
Common tasks#
| Task | Typical output | Main risk |
|---|---|---|
| Summarisation | Notes, action items | Unsupported or misattributed statements |
| Field extraction | JSON with names, dates, amounts | Plausible but wrong values |
| Classification | Contact reason, outcome, compliance flag | Label drift without a test set |
| Question answering | Answer with quoted evidence | Answers from world knowledge, not the call |
| Voice agent reasoning | Next reply, tool calls | Acting on a misheard request |
Structured output with evidence#
For anything machine-readable, define a schema and require the model to fill it. Many LLM APIs can constrain output to a JSON schema. Add an evidence field per value, such as the timestamp or a short quote, and validate that the quote occurs in the transcript. A value with no matching evidence is rejected or sent for review.
Long recordings and many recordings#
For one long meeting, a full transcript usually fits in context. Liu et al. (2023) found that models retrieve information from the middle of long inputs less reliably than from the start or end. A two-step approach, first extracting decisions, figures and actions with timestamps, then reasoning over that shorter list, is more reliable than one large prompt.
For questions across thousands of calls ("why did customers mention cancellations in March?"), use retrieval-augmented generation (Lewis et al., 2020). Chunk transcripts by turn or topic, index them with embeddings and metadata such as date and agent, retrieve the relevant chunks, and pass only those to the model. Keep call identifiers and timestamps in every chunk so answers can cite their source.
Privacy and safety#
- Treat transcripts as untrusted input. Separate instructions from transcript content clearly, never give the model permissions based on what a caller said, and validate outputs against rules.
- Minimise personal data. Redact what the task does not need before the prompt, and check where the LLM provider processes and stores data. Under GDPR the transcript is usually personal data; each processor needs a data processing agreement.
- Keep the source. An LLM output is derived data. Retain the ability to trace each claim to the transcript.
How Reson8 approaches this#
Reson8 provides the transcript layer, not the language model. The features most relevant here target the words LLMs cannot repair: text-only domain adaptation, which accepts up to 1M tokens of domain text at runtime without retraining, and Patterns for identifiers such as order numbers. For voice agents, the Turns endpoint is measured on semantic WER (1.10% pooled on the Pipecat benchmark), since that is closer to what the downstream LLM experiences. Audio is processed on EU GPUs with zero retention.
Common questions#
Should I send audio directly to a multimodal model instead?
For some tasks it works. A separate transcript is still easier to store, search, redact, audit and reuse across several downstream tasks, and specialist ASR is usually more accurate on domain terms.
Can the LLM fix transcription errors?
It can smooth obvious ones from context. It cannot reliably fix wrong digits, names or dropped negations, and it will not tell you when it guessed.
How do I test the combined system?
Build a set of recordings with human-verified transcripts and expected outputs. Run both the real transcript and the reference transcript through the LLM step. The difference tells you how much of the error comes from ASR and how much from the prompt.
References#
- Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS.
- Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
- Greshake, K. et al. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. ACM Workshop on Artificial Intelligence and Security (AISec).
- Maynez, J. et al. (2020). On Faithfulness and Factuality in Abstractive Summarization. ACL.
- Bain, M. et al. (2023). WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. Interspeech.
- Regulation (EU) 2016/679 (General Data Protection Regulation). EUR-Lex.