Voice agent frameworks

Voice agent frameworks provide the orchestration layer that sits between the caller and the models: transports for WebRTC, WebSockets and telephony, a streaming pipeline, turn detection, interruption handling and plug-ins for STT, LLM and TTS providers. The two most widely used open-source options are Pipecat and LiveKit Agents. They save months of work on real-time plumbing, and the choice between them depends mostly on transport, deployment model and how much structure your conversations need.

What a framework does#

A cascaded voice agent looks simple on a whiteboard. In production it needs to receive audio at a steady frame rate, run VAD, stream to STT, assemble transcripts into LLM context, stream tokens into TTS, play audio at the right pace, and tear all of that down within milliseconds when the user interrupts. Every provider has a different streaming protocol and error behaviour.

Concretely, frameworks handle:

• Transports. WebRTC for browsers and apps, WebSockets for server-to-server audio, and SIP or telephony provider media streams for phone calls.
• Streaming and back-pressure. Moving audio and text between stages without blocking, and pacing output audio to real time.
• Turn management. Running a VAD and a turn model, or consuming turn events from the STT provider.
• Interruptions. Cancelling in-flight LLM and TTS work, flushing buffers and updating the context.
• Context aggregation. Building the chat history from user transcripts and what the agent actually said.
• Observability. Metrics for time to first byte per service, and hooks for tracing.

Pipecat#

Pipecat is an open-source Python framework started by Daily. An agent is a pipeline of frame processors. Frames carry audio, text, LLM messages and control signals such as the start of user speech or an interruption. Each processor handles the frames it cares about and passes the rest along, and control frames can travel upstream as well as down.

An abridged pipeline, following the pattern in Pipecat's examples (exact class names vary between versions):

pipeline = Pipeline([
    transport.input(),              # caller audio in
    stt,                            # streaming speech-to-text
    context_aggregator.user(),      # add user turn to context
    llm,                            # stream reply tokens
    tts,                            # stream synthesised audio
    transport.output(),             # audio out to caller
    context_aggregator.assistant(), # record what was spoken
])

The linear structure makes the data flow explicit and easy to extend: insert a processor to redact PII, log transcripts or route by language. Pipecat supports many STT, LLM and TTS services, several transports including Daily's WebRTC and plain WebSockets, and telephony providers through their media stream protocols. Pipecat Flows adds structured, state-machine conversations for tasks such as intake forms. The Pipecat project also maintains Smart Turn, an open audio-based end-of-turn model, and a public STT benchmark for voice agents.

LiveKit Agents#

LiveKit Agents is an open-source framework from LiveKit, the company behind an open-source WebRTC media server. Agents join LiveKit rooms as participants. A worker process registers with the LiveKit server and is dispatched jobs as sessions start, which gives a natural model for scaling many concurrent calls. Within a session, you configure STT, LLM, TTS, VAD and turn detection, and define the agent's instructions and tools.

Because it sits on a media server, LiveKit Agents is a strong fit when the agent is one participant among several, for example in a video call, or when you already use LiveKit for real-time media. LiveKit also offers SIP integration for phone calls and its own turn detection model. It provides Python and Node.js SDKs.

Choosing#

Other open-source options include Vocode, and some LLM providers now ship voice helpers in their own SDKs. Managed platforms such as Vapi and Retell run the whole stack and expose it through a dashboard and API, which is convenient for prototypes and for teams without real-time infrastructure experience.

Evaluating a framework#

• Interruption quality. Barge in repeatedly on a long reply and check that audio stops promptly and the context records only what was played.
• Turn detection options. Can you use a silence timer, a local turn model, or turn events from the STT provider, and switch between them?
• Telephony path. If calls come from the phone network, test the actual carrier or SIP route, including 8 kHz audio. See voice agents on the phone.
• Metrics. Per-stage latency out of the box saves you building the latency budget instrumentation yourself.
• Testing hooks. Can you drive the pipeline from recorded audio or text in CI? See testing voice agents.
• Deployment. Where does the agent run, and can it run in the same region as your STT, LLM and TTS providers?

How Reson8 approaches this#

Reson8's Turns endpoint is a streaming STT service designed to sit inside these frameworks, providing transcripts, eager end-of-turn prediction and barge-in detection over one connection. On the open Pipecat STT benchmark (1,000 samples, smart-turn-data-v3.1) it recorded 1.10% pooled semantic WER and a 326 ms median time from end of speech to final transcript. Integration details are in the documentation.

Common questions#

References#

1. Pipecat: open-source framework for voice and multimodal conversational AI. GitHub.
2. Pipecat documentation. docs.pipecat.ai.
3. LiveKit Agents. GitHub.
4. LiveKit Agents documentation. docs.livekit.io.
5. Pipecat Smart Turn. GitHub.
6. Vocode. GitHub.