Text-to-speech (TTS) turns written text into spoken audio. A modern system normalises the text, works out how each word is pronounced and how the sentence should sound, then uses neural models to generate a waveform. In a voice agent it is the last stage of the pipeline, and its time to first audio matters as much as how natural it sounds.
Key takeaways
- TTS is the mirror image of speech recognition: text in, audio out. Most systems split into a text frontend, an acoustic model and a vocoder, though recent models merge these.
- The frontend decides pronunciation. "Dr." can be "doctor" or "drive"; "1/2" can be "a half" or "the first of February". Errors here sound confident and wrong.
- For voice agents, streaming matters more than peak quality. A voice that starts speaking 200 ms sooner often feels better than a slightly more natural one.
- Quality is judged by listeners (MOS, MUSHRA) and by round-trip intelligibility: synthesise, transcribe with ASR, and compute word error rate.
- Voice cloning raises consent and disclosure questions. The EU AI Act requires synthetic audio to be marked as AI-generated.
How text becomes speech#
A classic neural TTS system has three stages. Each one solves a different problem, and each can fail in its own way.
Text frontend#
The frontend is text processing, and it is the inverse of inverse text normalisation in ASR. It expands numbers, dates, currencies and abbreviations into words ("€20" becomes "twenty euros"), resolves ambiguous tokens from context, and then runs grapheme-to-phoneme conversion to get a pronunciation for each word. Dictionaries cover common words; a learned G2P model guesses the rest. It also predicts prosody: where to pause, which word to stress, whether a sentence rises as a question.
Acoustic model and vocoder#
The acoustic model maps phonemes to a time-frequency representation, usually a mel spectrogram, deciding duration and pitch for every sound. The vocoder turns that spectrogram into samples. WaveNet (2016) showed that a neural network could generate raw audio sample by sample at far higher quality than earlier methods, but it was slow. Later vocoders such as HiFi-GAN generate audio much faster than real time on a GPU.
Four generations of TTS#
| Approach | How it works | Strengths | Weaknesses |
|---|---|---|---|
| Formant / rule-based | Hand-written rules drive a model of the vocal tract | Tiny, predictable, intelligible at high speed | Clearly robotic |
| Concatenative (unit selection) | Stitches recorded fragments from a large single-speaker database | Natural on in-domain sentences | Audible joins, huge databases, one voice per recording effort |
| Neural two-stage | Tacotron 2 or FastSpeech predicts spectrograms; a neural vocoder renders audio | Near-human naturalness on read text | Needs hours of clean studio data per voice |
| Codec language models | Predicts discrete audio codec tokens, like an LLM predicts words | Zero-shot voices from a short prompt, expressive | Can skip or repeat words; harder to control |
The codec language model family, introduced prominently by VALL-E in 2023, treats speech as a sequence of tokens from a neural audio codec. Given a few seconds of someone's voice as a prompt, it continues in that voice. The same property that makes these models flexible also makes them prone to the TTS equivalent of hallucination: dropped words, repeated phrases, or a sudden change in speaking style.
TTS inside a voice agent#
In a cascaded voice agent, TTS receives text from the language model and must start speaking before the full reply exists. Three properties decide whether it works in conversation.
Streaming and time to first audio#
Agents stream LLM tokens into TTS in chunks, often a clause or sentence at a time, because prosody needs some look-ahead. Chunk too small and the voice sounds choppy; chunk too large and the caller waits. Time to first audio is one line in the latency budget, alongside end-of-turn detection, final transcript and the model's first token.
Interruption#
When a caller talks over the agent, playback must stop within a few hundred milliseconds, and the agent must know which words were actually heard. If the LLM believes it said the whole reply, it will refer to information the caller never received. Many TTS APIs return word-level timestamps so the orchestrator can truncate the conversation history at the point of barge-in.
Output format#
TTS models commonly generate 22.05, 24 or 44.1 kHz audio. A phone line carries 8 kHz G.711, so the audio is resampled and companded before it leaves. Test voices at the sample rate your callers will hear; a voice that sounds warm on studio monitors can sound thin through a handset.
Measuring TTS quality#
There is no single number for TTS as there is word error rate for ASR. Evaluations usually combine several.
- Mean opinion score (MOS). Listeners rate samples from 1 (bad) to 5 (excellent), following the absolute category rating method in ITU-T P.800. It is cheap to run and noisy; scores are only comparable within one listening test.
- MUSHRA. Listeners compare several systems side by side against a hidden reference (ITU-R BS.1534). Better at separating close systems.
- Round-trip intelligibility. Synthesise a test set, transcribe it with an ASR model, and compute WER against the input text. This catches skipped and repeated words that listeners might miss in a short clip.
- Speaker similarity. For cloned voices, the cosine similarity between speaker embeddings of the prompt and the output.
- Pronunciation checks. A targeted list of names, brands and codes your agent will actually say. General benchmarks rarely include them.
SSML, the W3C Speech Synthesis Markup Language, gives some control over the result: pauses, emphasis, phonetic spellings and interpretation hints such as dates or telephone numbers. Support varies by engine, and neural voices often ignore prosody tags they were not trained on.
Voice cloning, consent and disclosure#
A cloned voice is built from recordings of a real person, and a voice is personal data under the GDPR when the person is identifiable (see GDPR and voice data). Reputable providers require evidence of the speaker's consent before creating a clone.
The EU AI Act adds transparency duties. Article 50(2) requires providers of AI systems that generate synthetic audio to mark the output in a machine-readable format so it can be detected as artificially generated. Article 50(1) requires systems that interact with people to inform them they are dealing with an AI, unless it is obvious. These obligations apply from 2 August 2026. For a voice agent, the practical minimum is a spoken disclosure at the start of the call.
How Reson8 approaches this#
Reson8 does not offer text-to-speech. It provides the speech-to-text side of a voice agent, and the Turns endpoint supplies the two signals that decide when TTS should start and stop: eager end-of-turn prediction and native barge-in detection. Most teams pair it with a separate TTS provider through a framework such as Pipecat or LiveKit Agents (see voice agent frameworks).
Common questions#
Is TTS the same as speech synthesis?
Yes. Speech synthesis is the broader term and includes generating speech from representations other than text, but in practice the two are used interchangeably.
Why does TTS mispronounce names?
Names are often missing from the pronunciation dictionary, so a G2P model guesses from spelling, usually with the rules of the voice's language. A Dutch surname read by an English voice will follow English spelling rules. Phonetic overrides or a custom lexicon fix this for known names.
Do speech-to-speech models replace TTS?
They generate audio directly from audio, so there is no separate TTS stage. They can reduce latency, but they give up some of the control and inspectability of a text step. See speech-to-speech models vs pipelines.
What sample rate should a voice agent use?
Whatever the output channel carries. For telephony that is 8 kHz, and generating at a higher rate then resampling is normal. For web and app agents, 24 kHz is a common default.
References#
- Klatt, D. H. (1987). Review of text-to-speech conversion for English. Journal of the Acoustical Society of America, 82(3).
- van den Oord, A. et al. (2016). WaveNet: A Generative Model for Raw Audio. arXiv:1609.03499.
- Shen, J. et al. (2018). Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. ICASSP. arXiv:1712.05884.
- Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. NeurIPS. arXiv:2010.05646.
- Kim, J., Kong, J., & Son, J. (2021). Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (VITS). ICML. arXiv:2106.06103.
- Wang, C. et al. (2023). Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E). arXiv:2301.02111.
- W3C (2010). Speech Synthesis Markup Language (SSML) Version 1.1. W3C Recommendation.
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 50. EUR-Lex.