Most voice agents that answer real customers do it over the telephone network. Calls arrive through a carrier or a cloud telephony provider, usually as SIP signalling with G.711 audio at 8 kHz, and reach the agent either as RTP packets or as a WebSocket media stream. Narrowband audio, extra network hops, echo and keypad input all shape the design, and recognition accuracy on phone audio is measurably lower than on the wideband recordings most models are benchmarked on.
Key takeaways
- Phone audio is usually 8 kHz G.711 (μ-law in North America and Japan, A-law in most other regions), carrying roughly 300–3,400 Hz of speech bandwidth.
- Upsampling to 16 kHz makes the format acceptable to a model but does not restore the lost high frequencies. Evaluate STT on real call audio.
- Telephony adds latency before any model runs: carrier routing, jitter buffers and a media relay to your server.
- Plan for phone-specific features: DTMF input, call transfer, voicemail detection, hold music and recording disclosures.
- Barge-in on the phone depends on buffers you may not control. Keep outbound audio chunks small so playback can stop quickly.
The call path#
A call from a mobile or landline travels through the caller's carrier and the public switched telephone network (PSTN) to a phone number you rent from a telephony provider. The provider terminates the call and hands it to your system in one of two ways.
SIP and RTP#
The Session Initiation Protocol (RFC 3261) sets up, modifies and ends calls. Audio travels separately in RTP packets (RFC 3550), typically carrying 20 ms of audio each. Connecting directly over a SIP trunk gives you full control and the fewest hops, but you need a SIP stack or a media server, a public address, and handling for NAT, codecs and retransmissions. Frameworks such as LiveKit provide SIP integration for this reason.
WebSocket media streams#
Cloud telephony providers such as Twilio, Telnyx and Vonage can relay the call's audio to your server over a WebSocket. Twilio's Media Streams, for example, sends base64-encoded 8 kHz μ-law audio in JSON messages and accepts audio back in the same format. This is the fastest way to get started and is what many voice agent frameworks integrate with out of the box.
8 kHz reality#
G.711 samples at 8 kHz, which by the Nyquist limit can represent frequencies up to 4 kHz, and telephone channels are traditionally filtered to roughly 300–3,400 Hz. Much of the energy that distinguishes fricatives such as /s/, /f/ and /θ/ lies above that band. Humans fill the gap from context; models trained mostly on 16 kHz audio do less well.
The effect shows up on exactly the tokens agents need. Spelled letters are a well-known problem: F as in Freddie exists because F and S are hard to tell apart on a phone line. A caller spelling a postcode or licence plate produces S and the transcript reads F.
| Codec | Sample rate | Where you meet it |
|---|---|---|
| G.711 μ-law / A-law | 8 kHz | Default on PSTN and most SIP trunks |
| G.729 | 8 kHz | Low-bandwidth VoIP; lossy compression on top of narrowband |
| G.722 | 16 kHz | "HD voice" desk phones and some SIP trunks |
| AMR-WB | 16 kHz | Mobile HD voice; usually transcoded to G.711 at the network edge |
| Opus | Up to 48 kHz | WebRTC and app-based calling |
Even when both callers have HD-capable phones, the call usually crosses a network boundary that transcodes it to G.711. Assume narrowband unless you control both ends. For recorded calls rather than live agents, see telephony transcription, and for format details audio formats for speech recognition.
Latency, buffers and echo#
Telephony consumes part of the latency budget before the agent does anything. Carrier routing, the provider's media relay, jitter buffers at each hop and the WebSocket to your server all add delay in each direction. ITU-T G.114 treats up to 150 ms of one-way delay as acceptable for most interactive use; a voice agent's model processing comes on top of that.
Buffers also affect barge-in. If the agent sends several seconds of TTS audio to the provider at once, it will keep playing after the agent decides to stop unless the provider offers a way to clear queued audio. Stream in small chunks and use the provider's clear or flush command when interrupted.
Line echo from analogue hybrids is handled by network echo cancellers, but acoustic echo from a handset on speaker is not always fully removed. The agent can then hear its own voice and interrupt itself. Test on speakerphone and in a car.
Phone-specific features#
- DTMF input. Let callers type account numbers or PINs on the keypad. It is more accurate than speech for long digit strings and keeps some sensitive data out of the transcript.
- Call transfer. Hand the call to a human, either blind or with a warm introduction. SIP uses REFER (RFC 3515) or a provider API.
- Voicemail and answering-machine detection for outbound calls, so the agent leaves a message instead of starting a conversation with a greeting.
- Hold and music. Music on hold is non-speech audio that VAD may misread. Pause recognition during hold.
- Recording and disclosures. Many jurisdictions and industries require telling callers that a call is recorded or that they are speaking with an automated system. The EU AI Act includes transparency obligations for AI systems that interact with people; see the EU AI Act and speech recognition.
- Silence and hang-up handling. Detect when the caller has gone, and end the session to stop paying for idle compute.
How Reson8 approaches this#
Reson8's Turns endpoint provides streaming STT with eager end-of-turn prediction and native barge-in detection, which covers the recognition and turn-taking side of a phone agent. For identifiers read over the phone, such as the licence plates and order numbers above, Patterns biases recognition toward a declared format: on licence plates it lowered WER from 6.6% to 4.7%, and it corrected examples such as "W53TYN" to WU53 TYN. All audio is processed on EU GPUs with zero retention, which matters for call audio under GDPR.
Common questions#
Should I use SIP directly or a WebSocket media stream?
A media stream is quicker to build and fine for many deployments. Direct SIP removes a relay hop and gives more control over codecs and call handling, at the cost of running telephony infrastructure.
Does upsampling 8 kHz audio improve recognition?
It makes the audio compatible with a 16 kHz model, but adds no information. Models trained or adapted on telephony audio do better than upsampling alone.
How do I test a phone agent?
Call it through the real carrier path, from real phones, including mobiles on poor signal and speakerphone. Simulated calls over WebSocket miss codec loss and network delay. See testing voice agents.
References#
- Rosenberg, J. et al. (2002). SIP: Session Initiation Protocol. RFC 3261, IETF.
- Schulzrinne, H. et al. (2003). RTP: A Transport Protocol for Real-Time Applications. RFC 3550, IETF.
- Schulzrinne, H., & Taylor, T. (2006). RTP Payload for DTMF Digits, Telephony Tones, and Telephony Signals. RFC 4733, IETF.
- Sparks, R. (2003). The Session Initiation Protocol (SIP) Refer Method. RFC 3515, IETF.
- ITU-T Recommendation G.711 (1988). Pulse code modulation (PCM) of voice frequencies. International Telecommunication Union.
- ITU-T Recommendation G.114 (2003). One-way transmission time. International Telecommunication Union.
- Twilio. Media Streams. Twilio documentation.