Word timestamps give the start and end time of every recognised word. They drive captions, click-to-seek transcripts, audio redaction and speaker attribution. Timestamps come either from the recogniser's own decoding or from forced alignment, a separate step that places a known transcript on the audio timeline with an acoustic model. Precision is limited by the model's frame rate and by how the model was trained, and it varies more between systems than accuracy does.
Key takeaways
- A timestamp is only as precise as the model's output frames, typically 20 to 40 ms, and often worse at word boundaries.
- Forced alignment solves an easier problem than recognition: the words are known, only their positions are not. It is the standard way to get accurate timing.
- CTC models are "peaky": they fire each token in one frame, so start times are usually better than end times.
- Attention encoder-decoder models do not produce timestamps naturally. Timings are derived from attention weights or a separate aligner.
- Check timing on your own audio. Two systems with similar WER can differ by hundreds of milliseconds at word boundaries.
What timestamps are used for#
- Captions and subtitles. Cue boundaries come from word times. See captions and subtitles.
- Interactive transcripts. Click a word, jump to that point in the recording. Legal and medical reviewers rely on this to check a transcript against the audio.
- Audio redaction. To bleep a card number out of a call recording, you need to know exactly when it was spoken. See PII redaction.
- Speaker attribution. Diarization gives time ranges; timestamps map each word into one of them.
- Analytics. Talk-time ratios, pauses, speaking rate and interruptions all come from word times.
- Training data. Long recordings with a transcript are cut into short training segments by aligning them first.
Forced alignment#
Forced alignment takes audio and a transcript that is already correct, from a human or a recogniser, and finds when each word occurs. The acoustic model computes, for every frame, a probability for each token. Alignment then finds the single path through a grid of frames and transcript tokens that maximises total probability, subject to the constraint that tokens appear in order.
HMM-based aligners#
The classic approach uses a hidden Markov model acoustic model and a pronunciation lexicon, with Viterbi decoding restricted to the known word sequence. The Montreal Forced Aligner, built on Kaldi, is widely used in phonetics research and gives phone-level as well as word-level boundaries. It needs a pronunciation dictionary for the language, or a grapheme-to-phoneme model to generate one.
CTC-based aligners#
Neural CTC models can be used the same way, aligning characters or subword tokens instead of phones. CTC segmentation, described by Kürzinger and colleagues, aligns long recordings with their transcripts and was used to build German training corpora. Meta's MMS work used CTC alignment at scale across more than a thousand languages, and torchaudio ships an aligner based on it. No lexicon is needed, which makes it easy to apply to new languages.
The dynamic programme#
Stripped of blanks and details, alignment is a short dynamic programme. Given frame-level log-probabilities and the token sequence, it finds where each token starts:
import numpy as np
def align(log_probs, tokens):
"""log_probs: (T, V) per-frame log-probabilities. tokens: L ids, L <= T.
Returns the start frame of each token on the best monotonic path."""
T, L = len(log_probs), len(tokens)
score = np.full((T, L), -np.inf)
advanced = np.zeros((T, L), dtype=bool)
score[0, 0] = log_probs[0, tokens[0]]
for t in range(1, T):
stay = score[t - 1]
move = np.concatenate(([-np.inf], score[t - 1, :-1]))
advanced[t] = move > stay
score[t] = np.maximum(stay, move) + log_probs[t, tokens]
starts, j = [0] * L, L - 1
for t in range(T - 1, 0, -1):
if advanced[t, j]:
starts[j] = t
j -= 1
return starts
Multiply start frames by the frame duration to get seconds. Real CTC aligners add a blank state between tokens, which is where silence and the gaps between peaky emissions go.
Timestamps from the recogniser itself#
Most speech-to-text APIs return word times directly. How they are produced depends on the architecture:
| Architecture | How timing is obtained | Typical weakness |
|---|---|---|
| Hybrid HMM | Directly from the decoding path | Needs a lexicon; good boundaries |
| CTC | Frame of each token emission | Peaky spikes; end times imprecise |
| RNN-T | Frame at which each token is emitted | Emission can lag the audio, shifting times late |
| Attention encoder-decoder | Attention weights plus alignment, or a second-pass aligner | No native timing; drift on long audio |
Whisper, an encoder-decoder, predicts coarse segment timestamps as tokens. Its word-level option aligns words by applying dynamic time warping to cross-attention weights. WhisperX takes a different route: it transcribes with Whisper, then runs a separate CTC phoneme model to force-align the words, which gives tighter boundaries.
What limits timestamp quality#
- Frame rate. An encoder that outputs one frame per 40 ms cannot place a boundary more precisely than that.
- Word boundaries in speech. Connected speech has no silence between most words. "Want to" becomes "wanna"; where one word ends is partly a convention.
- Transcript errors. Alignment trusts the transcript. A missing or extra word pushes neighbouring words out of place until the path recovers.
- Long audio. Small drifts accumulate. Aligning in chunks anchored at confident words or silences keeps errors local. See long-form transcription.
- Non-speech. Music, laughter and noise inside a segment can absorb or displace words.
- Normalised text. "€20" has to be aligned as the spoken "twenty euros". Align the spoken form, then map back to the written form. See inverse text normalisation.
Common questions#
How accurate are word timestamps?
For good aligners on clean speech, most word starts fall within a few tens of milliseconds of hand labels. Recogniser-native timestamps can be considerably looser, especially for end times and in noisy or fast speech.
Do I need forced alignment if my API returns timestamps?
Only if the returned times are not precise enough for your use. Captions and seek-to-word usually tolerate them. Audio redaction and phonetic analysis often justify a dedicated aligner.
Can I align a human transcript to audio?
Yes. That is the original use of forced alignment. Clean verbatim transcripts align best; edited transcripts with removed fillers or reordered sentences need an aligner that tolerates mismatches.
References#
- McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M., & Sonderegger, M. (2017). Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. Interspeech.
- Kürzinger, L., Winkelbauer, D., Li, L., Watzel, T., & Rigoll, G. (2020). CTC-Segmentation of Large Corpora for German End-to-End Speech Recognition. SPECOM.
- Bain, M., Huh, J., Han, T., & Zisserman, A. (2023). WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. Interspeech.
- Pratap, V. et al. (2023). Scaling Speech Technology to 1,000+ Languages. arXiv:2305.13516.
- Zeyer, A., Schlüter, R., & Ney, H. (2021). Why does CTC result in peaky behavior? arXiv preprint.
- Radford, A. et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356.