How speech-to-text works

Speech-to-text works in four stages. The audio is cut into short frames and turned into a log-mel spectrogram. A neural encoder converts those frames into vectors that describe what is being said in context. A decoder searches for the most probable sequence of text tokens given those vectors. Finally, post-processing adds punctuation, casing, number formatting and timestamps so the output reads like a transcript rather than a string of tokens.

From audio to features#

Microphones produce a pressure wave; an analogue-to-digital converter samples it. Speech models usually expect 16 kHz mono audio, 16,000 amplitude values per second, which captures frequencies up to 8 kHz and so covers the information that matters for intelligibility. Telephone audio is 8 kHz and loses everything above 4 kHz, which is why fricatives such as "f" and "s" are hard to tell apart on calls. See audio formats.

The signal is then cut into overlapping frames, conventionally 25 ms long with a 10 ms step. For each frame, a short-time Fourier transform gives the energy at each frequency. Those energies are pooled into mel bands, which are spaced more densely at low frequencies, roughly as human hearing is, and then log-compressed. The result is a log-mel spectrogram: an image with time on one axis and frequency on the other.

import torch, torchaudio

waveform, sr = torchaudio.load("call.wav")          # shape: [channels, samples]
waveform = torchaudio.functional.resample(waveform, sr, 16000)

mel = torchaudio.transforms.MelSpectrogram(
    sample_rate=16000, n_fft=400, hop_length=160, n_mels=80
)(waveform)
log_mel = torch.log(mel + 1e-6)                      # [channels, 80, frames]
print(log_mel.shape)                                 # ~100 frames per second

Here n_fft=400 is 25 ms and hop_length=160 is 10 ms at 16 kHz. Older HMM systems went one step further and computed mel-frequency cepstral coefficients (MFCCs); neural models learn better from the log-mel bands directly. Some self-supervised models, such as wav2vec 2.0, skip hand-designed features and learn a convolutional front end from the raw waveform.

The encoder#

The encoder takes the feature frames and produces a sequence of vectors, one per short stretch of time, that represent the speech in context. Current encoders are mostly Transformers or Conformers, which add convolution to capture local patterns such as formant transitions alongside self-attention for longer-range context.

The first layers subsample in time, commonly by a factor of four so that each output vector covers 40 ms, and in some efficient designs by eight. This matters for cost, because attention scales with the square of sequence length, and it matches the rate of speech: syllables last on the order of 100 to 300 ms, so 100 frames per second is more resolution than the output needs.

The decoder#

The decoder turns encoder vectors into text. Three designs dominate, compared in detail in ASR model architectures:

• CTC predicts a token or a "blank" for every encoder vector independently, then collapses repeats and blanks. It is fast and simple, but each prediction ignores the others, so it relies on the encoder or an external language model for fluency.
• Transducers (RNN-T and its variants) add a small prediction network that conditions each output on the tokens already emitted. They stream naturally and are common in real-time systems.
• Attention encoder-decoders generate tokens one at a time, attending over all encoder vectors, like a translation model. They are accurate on complete utterances and harder to stream.

Tokens#

Outputs are sub-word tokens from a vocabulary of a few hundred to tens of thousands of units, learned with byte-pair encoding or a unigram model such as SentencePiece. Common words are single tokens; rare words are spelled from pieces. This lets a model write a surname it has never heard, but it also means it will happily write a plausible wrong spelling: Nguyen becomes Win.

Search#

Picking the single most probable token at each step, greedy decoding, is fast and often good enough. Beam search keeps several candidate transcripts and extends each, which helps when an early choice only makes sense given later words. Decoding is also where external knowledge is added:

• An external language model can rescore candidates, known as shallow fusion.
• Context biasing boosts the probability of expected words or phrases.
• Format constraints steer output toward a known pattern for identifiers.

Post-processing#

The decoder's output is a sequence of tokens. Turning it into a usable transcript takes several more steps, some of which modern models perform internally:

Models trained on web-scale transcripts, such as Whisper, learned to output punctuated, cased and partly formatted text directly, because their training targets looked like that. Others emit lower-case spoken form and rely on a separate formatting model.

How the model learns#

A speech model is trained on pairs of audio and transcript. The loss depends on the architecture: CTC loss and transducer loss sum over all possible alignments of the text to the audio, so no one has to mark where each word starts; attention decoders use cross-entropy on the next token. Training data runs from thousands to hundreds of thousands of hours, and the variety of that data, accents, microphones, rooms and topics, matters more than raw volume for how well the model holds up on real-world audio.

Many recent systems add stages beyond supervised training: self-supervised pre-training on untranscribed audio, fine-tuning on curated high-quality sets, and optimisation against sequence-level objectives such as word error rate itself. The latter borrows from reinforcement learning, because WER cannot be differentiated directly.

How Reson8 approaches this#

Resonant-1 was trained on a four-phase schedule that relies heavily on reinforcement learning, and is served on Reson8's own inference engine, Violin, which uses fused kernels, CUDA graphs and decoding scheduling to keep GPU inference fast. Domain terms are handled at decoding time: text-only domain adaptation accepts up to 1M tokens of context applied at runtime, and Patterns bias output toward an expected identifier format.

Common questions#

References#

1. Davis, S., & Mermelstein, P. (1980). Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4).
2. Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006). Connectionist temporal classification. ICML.
3. Graves, A. (2012). Sequence Transduction with Recurrent Neural Networks. arXiv:1211.3711.
4. Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016). Listen, Attend and Spell. ICASSP.
5. Gulati, A. et al. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. Interspeech.
6. Kudo, T., & Richardson, J. (2018). SentencePiece: a simple and language independent subword tokenizer and detokenizer. EMNLP.
7. Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: a framework for self-supervised learning of speech representations. NeurIPS.