Turn-taking is the set of rules that decides who speaks when. For a voice agent it has two halves: knowing when the user has finished, and knowing when the user wants the agent to stop. Barge-in is the second half. A good agent stops promptly when interrupted, ignores "mm-hm" and background noise, and remembers only the part of its reply the user actually heard.
Key takeaways
- Not every sound during agent speech is an interruption. Backchannels, coughs, echo and a television in the background all trigger naive detectors.
- Stopping fast matters. If the agent keeps talking for a second after the user starts, the user raises their voice or gives up.
- After stopping, trim the conversation history to what was played. Otherwise the LLM believes the user heard a sentence they never did.
- Echo cancellation is a prerequisite. Without it, the agent hears itself and interrupts its own replies.
- Some content should be non-interruptible, such as a legally required disclosure, and the policy needs to be explicit.
How humans take turns#
Conversation runs with very little overlap and very little silence. Sacks, Schegloff and Jefferson's 1974 model describes speakers projecting where a turn will end and timing their entry to that point. Overlap does occur, but it is usually brief and gets resolved within a syllable or two, as Schegloff analysed in detail in 2000.
Listeners also speak without taking the turn. Yngve called these short responses the "back channel": "mm-hm", "right", "ja", "oui". They signal attention and agreement, and a speaker carries on through them. A voice agent that stops every time the user says "yeah" feels jumpy and hard to listen to.
The two jobs of a turn manager are therefore:
- Taking the turn when the user has finished. See end-of-turn detection.
- Yielding the turn when the user wants to speak, and not yielding when they are only acknowledging.
What happens during a barge-in#
A barge-in handler has to do four things, in order:
- Detect that the user has started speaking with intent to take the turn.
- Stop TTS playback and flush every audio buffer downstream: the TTS stream, the orchestrator's queue, the transport and, if possible, the client's jitter buffer.
- Cancel any in-flight LLM generation and pending TTS requests for the old reply.
- Reconcile the conversation history, keeping only the words that were played, then process the user's new turn.
The time from the user's first syllable to silence from the agent is the barge-in latency. Buffers are the usual problem: an agent can decide to stop in 100 ms and still play half a second of audio already queued in a telephony gateway.
False barge-ins#
An agent that stops at any detected sound will stop constantly. The main sources:
| Source | Why it triggers | Mitigation |
|---|---|---|
| Echo of the agent's own voice | Speakerphone or poor handset isolation feeds playback into the mic | Acoustic echo cancellation on the client, or line echo cancellation in the network |
| Backchannels | "Mm-hm", "okay", "ja" are real speech | Require a minimum duration or word count; classify short acknowledgements |
| Background speech | Television, colleagues, a car radio | Speaker-aware VAD, noise suppression, a higher energy threshold during playback |
| Non-speech noise | Coughs, door slams, keyboard | A neural VAD rather than an energy threshold |
Echo deserves special attention on the phone. Line echo from hybrid circuits is handled by network echo cancellers of the type specified in ITU-T G.168, but acoustic echo from a mobile on speaker is left to the handset. Browser clients using WebRTC get acoustic echo cancellation from the browser's audio pipeline, provided it has not been disabled.
Interruption policies#
Whether to yield is a product decision as much as a detection problem. Common policies:
- Always interruptible. Stop on any confirmed user speech. Natural, but sensitive to false triggers.
- Minimum speech threshold. Yield only after, say, a few hundred milliseconds or two or more words of user speech. Filters backchannels at the cost of a slightly slower stop.
- Content-based. Transcribe the overlapping speech and yield only if it is not an acknowledgement. Accurate but slowest, since it waits for STT.
- Non-interruptible segments. Mark specific utterances, such as a recording disclosure or terms read aloud, as protected. Keep listening, buffer what the user says, and handle it once the segment ends.
Whatever the policy, the agent should not treat an interruption as a failure. A user who says "yes, yes, I know" while the agent reads a long confirmation is being efficient. Shorter replies reduce the number of interruptions in the first place.
The agent's side of turn-taking#
Turn-taking also covers what the agent does when it holds the floor:
- Short turns. Spoken replies should be a sentence or two. Long paragraphs invite interruption and are hard to follow by ear.
- Handing back the turn. End with a clear question when input is needed, so the user knows it is their turn.
- Silence handling. If the user says nothing for several seconds after a question, prompt once, then offer an alternative or close the call.
- Holding the floor during lookups. A brief "One moment" during a tool call tells the user the agent has not frozen.
How Reson8 approaches this#
Reson8's Turns endpoint includes native barge-in detection alongside eager end-of-turn prediction, so a single streaming connection reports both when the user has finished and when they have started speaking over the agent. The same stream delivers the transcript, so the orchestrator can apply a content-based policy without a second service.
Common questions#
How fast should an agent stop when interrupted?
As fast as the audio path allows. Aim for the agent to go quiet within a few hundred milliseconds of the user starting; much beyond that and users start talking louder or repeating themselves. Measure end to end, including telephony buffers.
Should the agent resume where it left off?
Rarely word for word. Let the LLM see what was heard and what the user said, and decide whether to continue, answer the new question or rephrase.
Why does my agent interrupt itself?
Almost always echo: the agent's own voice returns through the microphone and is detected as user speech. Check that echo cancellation is on, and test on speakerphone.
Can the agent use backchannels too?
It can, but done badly it sounds mechanical. Most production agents stay silent while listening and rely on quick responses instead.
References#
- Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4).
- Schegloff, E. A. (2000). Overlapping talk and the organization of turn-taking for conversation. Language in Society, 29(1).
- Yngve, V. H. (1970). On getting a word in edgewise. Papers from the Sixth Regional Meeting of the Chicago Linguistic Society.
- Ward, N., & Tsukahara, W. (2000). Prosodic features which cue back-channel responses in English and Japanese. Journal of Pragmatics, 32(8).
- Skantze, G. (2021). Turn-taking in conversational systems and human-robot interaction: a review. Computer Speech & Language, 67.
- ITU-T Recommendation G.168. Digital network echo cancellers. International Telecommunication Union.