Long-form transcription

Long-form transcription is the transcription of recordings that run for minutes or hours: meetings, hearings, lectures, podcasts, consultations. Few models can take an hour of audio in one pass, so the audio is split into segments, transcribed, and joined back together. Most long-form errors come from that process rather than from recognition itself: words lost or duplicated at boundaries, sentences cut in half, timestamps that drift, speakers relabelled from one segment to the next.

Why long audio has to be segmented#

Encoder-decoder models are trained on short clips. Whisper processes exactly 30 seconds of audio per window; audio longer than that is handled by sliding the window forward. Transformer attention cost grows with the square of the sequence length, so even models without a hard limit become slow and memory-hungry on hour-long inputs, and their accuracy often degrades on lengths far beyond what they saw in training. A 2019 comparison of end-to-end architectures by Chiu and colleagues at Google found that attention-based models in particular generalised poorly to utterances much longer than their training data, while frame-synchronous models such as RNN-T held up better.

Segmentation also brings practical benefits. Segments can be transcribed in parallel across GPUs, which is why batch transcription of a two-hour file can finish in well under a minute. And a failure in one segment, such as a repetition loop, is contained rather than derailing the rest of the file.

Segmentation strategies#

Fixed windows#

The simplest approach cuts every n seconds. It is predictable and trivially parallel, but boundaries land mid-word about as often as not. A cut through "pharmaco|kinetics" leaves each segment with half a word, and each half is transcribed as something else.

Overlapping windows#

Adding a few seconds of overlap between windows means every word appears whole in at least one segment. The cost is a merge problem: the overlap region is transcribed twice, possibly differently, and the two versions must be reconciled.

VAD-based segmentation#

A voice activity detector finds pauses, and segments are cut there, merged up to a target length. Boundaries fall between words, so no merging is needed, and non-speech is excluded, which reduces hallucinations. WhisperX popularised this approach for Whisper and reported both faster processing and fewer repetition and hallucination errors than the sliding-window default. Its weakness is continuous speech with no usable pauses, such as a fast talker or overlapping speakers, where a fallback cut is still required.

Stitching segments together#

Merging overlapping transcripts is easiest with word-level timestamps. A common rule keeps each word from the window in which it sits furthest from an edge, since models are least reliable near the start and end of their input. Without timestamps, the fallback is to align the two overlapping texts by string matching and cut at the longest common run of words.

def stitch(segments):
    """segments: list of (window_start, window_end, words); each word has
    'start', 'end' in absolute seconds. Windows are sorted and overlap."""
    merged = []
    for i, (w_start, w_end, words) in enumerate(segments):
        lo = w_start if i == 0 else (w_start + segments[i - 1][1]) / 2
        hi = w_end if i == len(segments) - 1 else (segments[i + 1][0] + w_end) / 2
        for word in words:
            mid = (word["start"] + word["end"]) / 2
            if lo <= mid < hi:
                merged.append(word)
    return merged

The cut point is the midpoint of each overlap, so every word is taken from the window where it is further from the edge. Real systems also handle words whose timestamps disagree slightly between windows, and punctuation that must be re-run across the join.

Carrying context across segments#

Whisper can condition each window on the text of the previous one. That keeps spelling of names and formatting consistent through a file, but it also lets errors propagate: a hallucination or loop in one window can be copied into the next. Many production pipelines disable it, or pass only a short, filtered context. Context biasing with a fixed list of names and terms gives consistency without that risk.

Keeping the whole file consistent#

• Timestamps. Each segment's timestamps must be offset by the segment's start time. Off-by-one errors in the offset, or resampling that changes duration, cause drift that grows through the file. Check the last minute of a long file, not just the first.
• Speakers. If diarization runs per segment, "Speaker 1" in one segment may be "Speaker 2" in the next. Run diarization on the whole recording, or link speaker embeddings across segments.
• Formatting. Inverse text normalisation and punctuation should see complete sentences, which means running them after stitching, not per segment.
• Silence and non-speech. Long recordings contain breaks, background music and people leaving the room. Exclude them before recognition rather than filtering text afterwards.

Evaluating long-form accuracy#

Most familiar benchmarks are short-form: utterances already segmented by a human, so segmentation errors never appear. Long-form evaluation scores the full, unsegmented recording against a full reference, which tests the whole pipeline. Datasets used for this include Earnings-21 and Earnings-22 (earnings calls), TED-LIUM in its long-form configuration, and others collected for the Whisper paper. The Open ASR Leaderboard now reports long-form English separately from short-form. Rankings on the two do not always agree, because a model's segmentation and stitching strategy contributes to the long-form score. See ASR benchmarks.

How Reson8 approaches this#

Reson8 supports batch transcription of pre-recorded audio alongside real-time streaming. On the long-form English sets of the Open ASR Leaderboard (April 2026), Resonant-1 averaged 8.58% WER, placing third in that table after ElevenLabs Scribe v2 (7.32%) and AssemblyAI Universal 3 Pro (8.34%). On short-form English it had the lowest average at 6.51%. The figures are in the benchmarks post.

Common questions#

References#

1. Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.
2. Bain, M., Huh, J., Han, T., & Zisserman, A. (2023). WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. Interspeech.
3. Chiu, C.-C. et al. (2019). A Comparison of End-to-End Models for Long-Form Speech Recognition. IEEE ASRU.
4. Del Rio, M. et al. (2021). Earnings-21: A Practical Benchmark for ASR in the Wild. Interspeech.
5. Del Rio, M. et al. (2022). Earnings-22: A Practical Benchmark for Accents in the Wild. arXiv:2203.15591.
6. Hernandez, F. et al. (2018). TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation. SPECOM.
7. Srivastav, V. et al. Open ASR Leaderboard. Hugging Face.