Speech recognition models are trained on audio of a particular shape, most often 16 kHz mono, and perform best when the input matches it. The format choices that matter are sample rate, which sets how much of the voice's frequency range survives; the codec, which decides what compression throws away; and channels, which can keep speakers apart for free. Get these right at capture time, because no later conversion can restore information that was never recorded.
Key takeaways
- 16 kHz, 16-bit, mono linear PCM is the de facto standard input for speech models. It covers frequencies up to 8 kHz, which holds nearly all of the information in speech.
- Upsampling 8 kHz telephone audio to 16 kHz makes it the right shape but adds no information. The missing high band stays missing.
- Lossless formats (WAV, FLAC) are safe. Lossy codecs are acceptable at speech-appropriate bitrates, but each re-encode compounds the damage.
- When each speaker is on a separate channel, transcribe the channels separately instead of mixing to mono.
- A container (WAV, Ogg, MP4) is not a codec (PCM, Opus, AAC). Check both.
Sample rate and bandwidth#
A digital recording samples the waveform at fixed intervals. By the Nyquist–Shannon sampling theorem, a sample rate of f can represent frequencies up to f/2. Human speech has most of its energy below 4 kHz, but fricatives such as /s/, /f/ and /θ/ carry distinguishing energy well above that. The difference between "fin" and "thin", or between "s" and "f" when someone spells a code, lives largely in that upper band.
This is why most speech models, from wav2vec 2.0 to Whisper, operate on 16 kHz input. Higher rates such as 44.1 or 48 kHz cost bandwidth and storage and are simply downsampled by the model's front end. Lower rates are the telephone world: narrowband telephony samples at 8 kHz and band-limits speech to roughly 300–3,400 Hz. See telephony transcription for how models cope with that.
Bit depth, levels and clipping#
16-bit samples give around 96 dB of dynamic range, more than any real speech recording needs. 24-bit and 32-bit float are fine as inputs but add nothing for recognition. What matters more is level. Audio recorded too quietly sits near the noise floor; audio recorded too hot clips, flattening the waveform peaks and adding distortion across the spectrum. Clipping cannot be undone. Aggressive automatic gain control and noise suppression on the capture device can also pump background noise up between words or chop word onsets, which are harder for a model to handle than steady noise. See transcribing noisy audio.
Codecs and containers#
A container is the file format: WAV (RIFF), Ogg, WebM, MP4/M4A, FLAC's native format. A codec is how the audio inside is encoded. A .wav file almost always holds linear PCM but can hold μ-law or other encodings; a .webm file from a browser usually holds Opus. APIs that accept "WAV" or "raw audio" need to know the encoding, sample rate and channel count, either from a header or from parameters you send.
| Encoding | Type | Typical use | Notes for ASR |
|---|---|---|---|
| Linear PCM (s16le) | Uncompressed | WAV files, raw streams | Reference quality; 256 kbit/s at 16 kHz mono |
| FLAC | Lossless | Archives, uploads | Identical to PCM after decoding, roughly half the size |
| Opus | Lossy | WebRTC, browsers, VoIP | Built for interactive speech and music; good from 16–32 kbit/s |
| AAC, MP3 | Lossy | Media, podcasts, video | Fine at music bitrates; degrades at very low ones |
| G.711 μ-law / A-law | Companded | Telephony, SIP | 8 kHz, 64 kbit/s; μ-law in North America and Japan, A-law in Europe |
| G.722, AMR-WB | Lossy wideband | HD voice | 16 kHz sampling, up to 7 kHz audio band |
Lossy codecs discard information a perceptual model judges inaudible. At speech-appropriate bitrates the effect on recognition is usually small. The real damage comes from transcoding chains: a call recorded as G.711, converted to MP3 at a low bitrate for storage, then re-encoded again for upload. Each step removes a little more, and the losses compound. Keep the original encoding when you can, and convert once, directly to what the recogniser wants.
Size and bandwidth#
Uncompressed 16 kHz, 16-bit mono PCM is 32,000 bytes per second, about 1.9 MB per minute or 115 MB per hour. For a streaming connection that is 256 kbit/s upstream per stream, easily handled by a server but significant on mobile or at thousands of concurrent calls. Opus at 24–32 kbit/s reduces that by close to an order of magnitude.
Channels#
Speech models transcribe a single channel. Given stereo, most systems downmix to mono, which is correct for a stereo microphone recording of one room. It is wrong for dual-channel call recordings, where the agent is on one channel and the customer on the other. Mixing them throws away perfect speaker separation and creates overlapping speech where both talk at once. Transcribe each channel independently and merge the results by timestamp; you get speaker labels without any diarization errors.
Multichannel microphone arrays, such as meeting-room devices, are a different case. Their channels record the same sound from different positions, and the useful step is beamforming to one enhanced channel before recognition, not separate transcription.
Recommended settings#
# Convert anything ffmpeg can read to 16 kHz, 16-bit mono PCM WAV
ffmpeg -i input.m4a -ac 1 -ar 16000 -c:a pcm_s16le output.wav
# Split a dual-channel call recording into one file per speaker
ffmpeg -i call.wav -filter_complex "channelsplit=channel_layout=stereo[L][R]" \
-map "[L]" -ar 16000 agent.wav -map "[R]" -ar 16000 customer.wav
- Capture at 16 kHz or higher, mono per speaker, 16-bit, with peaks well below full scale.
- Stream linear PCM or Opus in chunks of 20–100 ms, at the rate the API expects.
- Upload the original file if the API accepts it. Otherwise convert once, to FLAC or PCM.
- Telephony: send the native 8 kHz G.711 or PCM if supported, rather than upsampling yourself.
Common questions#
Will recording at 48 kHz improve accuracy?
Not measurably for most speech models, which resample to 16 kHz internally. It costs three times the bandwidth. Record at 48 kHz only if you need the audio for other purposes such as broadcast.
Should I upsample 8 kHz audio before sending it?
Only if the API cannot accept 8 kHz. Upsampling does not recover the missing band, and some systems apply telephony-specific handling when they know the input is narrowband.
Is MP3 good enough for transcription?
At typical bitrates of 64 kbit/s and above, yes, with little effect on accuracy. Very low bitrates and repeated re-encoding cause audible artefacts that do increase errors.
Does a stereo file get transcribed as two speakers?
Only if the API supports per-channel transcription and you enable it. Otherwise the channels are mixed to mono first.
References#
- Shannon, C. E. (1949). Communication in the Presence of Noise. Proceedings of the IRE, 37(1).
- ITU-T Recommendation G.711. Pulse code modulation (PCM) of voice frequencies. International Telecommunication Union.
- ITU-T Recommendation G.722. 7 kHz audio-coding within 64 kbit/s. International Telecommunication Union.
- Valin, J.-M., Vos, K., & Terriberry, T. (2012). Definition of the Opus Audio Codec. RFC 6716, IETF.
- Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. NeurIPS.
- FFmpeg documentation. ffmpeg.org.