Tool calling in voice agents

Tool calling lets a voice agent do more than talk: look up an order, check a balance, book an appointment or transfer the call. The language model emits a structured request, the application executes it and returns the result, and the model turns that result into a spoken reply. In voice, three things make this harder than in chat: every call adds silence the user can hear, arguments come from a speech transcript that may be wrong, and results have to be turned into something worth listening to.

How tool calling works#

The application sends the model a list of tools, each with a name, a description and a JSON Schema for its parameters. When the model decides a tool is needed, it replies with a tool call instead of, or before, text. The application runs the function, adds the result to the conversation, and calls the model again to produce the reply. The technique was popularised by research such as ReAct (Yao et al., 2023) and Toolformer (Schick et al., 2023), and all major LLM APIs now support it natively.

lookup_order = {
    "name": "lookup_order",
    "description": "Find an order by its order number. Use when the caller asks about delivery or status.",
    "parameters": {
        "type": "object",
        "properties": {
            "order_id": {
                "type": "string",
                "pattern": "^INV[0-9]{4}[A-Z]$",
                "description": "Order number, e.g. INV7625X. Spoken digits must be converted to numerals."
            }
        },
        "required": ["order_id"]
    }
}

Timing: the silence problem#

A tool call adds at least one external request and one extra model pass to the latency budget. A database lookup may take tens of milliseconds; a legacy booking system can take several seconds. In chat, a spinner covers the wait. On a call, the user hears nothing and may assume the line has dropped.

Common techniques:

• Spoken acknowledgement. Have the model, or the orchestrator, say a short phrase such as "Let me check that" when it starts a slow tool. Vary it, and skip it for fast tools.
• Parallel calls. Most APIs let the model request several tools at once. Run independent lookups concurrently.
• Prefetching. If the caller is identified from their phone number, load their account before they ask.
• Asynchronous tools. For long operations, start the job, tell the caller it is under way, and continue the conversation until the result arrives.
• Timeouts. Give every tool a deadline and a spoken fallback: "The system is slow right now, I'll send the details by text."

Arguments come from speech#

The model fills tool arguments from the transcript. If the transcript is wrong, the call is wrong, and the model will often not notice. A caller says an order number and the recogniser produces INB seven six two five X. The model either calls lookup_order("INB7625X"), gets "not found", and apologises for an order that exists, or guesses.

Defences, in the order they apply:

1. Recognise the value correctly. Tune the STT stage for the formats your tools accept. See structured identifiers and numbers and alphanumerics.
2. Normalise. Convert spoken forms to canonical ones: "twenty-third of March" to a date, "double seven" to 77. This is inverse text normalisation, whether done by the STT or by code before the tool runs.
3. Validate. Check arguments against the schema and business rules before executing. Return a clear error the model can act on, such as "order number must be INV followed by four digits and a letter", so it asks the caller again rather than failing silently.
4. Confirm. Read back critical values before actions that move money, change bookings or share personal data.

Speaking the results#

A tool result designed for a web page is rarely good speech. A delivery API that returns timestamps, carrier codes and a tracking URL should become "It left the warehouse this morning and should arrive on Thursday." Help the model by:

• Returning only the fields the conversation needs, with plain names.
• Pre-formatting values for speech where precision is not needed: "about €40" rather than "€39.97".
• Instructing the model never to read URLs, IDs or JSON aloud unless asked.
• Splitting long lists: offer the first two options and ask if the caller wants more.

Interruptions, idempotency and safety#

• Barge-in during a tool call. If the user interrupts with "actually, cancel that", the orchestrator must decide whether the tool already ran. Track tool state explicitly, and design actions to be cancellable or reversible. See turn-taking and barge-in.
• Idempotency. Retries and repeated turns can call the same tool twice. Use idempotency keys for payments, bookings and messages.
• Authentication before action. Verify the caller before any tool that reveals or changes personal data. Do not rely on the model to enforce this; gate it in code.
• Least privilege. Give the agent tools scoped to the task. A support agent that can issue refunds should have a limit enforced by the backend, not by the prompt.
• Injection through results. Text returned by tools, such as a customer note or an email body, can contain instructions. Treat tool output as data, and keep sensitive actions behind code-level checks.

The Model Context Protocol (MCP) has become a common way to expose tools to models through a standard interface. It simplifies integration, but the same rules apply: the agent can do anything the connected tools allow.

How Reson8 approaches this#

Reson8 addresses the argument problem at the recognition stage. With Patterns, you supply the identifier format a tool expects, such as INV[0-9]{4}[A-Z], and the model biases toward it. The spoken example above, "INB seven six two five X", is transcribed as INV7625X. On order numbers this reduced WER from 8.0% to 1.5%. Domain adaptation accepts product names and other domain text through the API, taking effect in under 60 seconds.

Common questions#

References#

1. Yao, S. et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR. arXiv:2210.03629.
2. Schick, T. et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS. arXiv:2302.04761.
3. Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045.
4. Greshake, K. et al. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173.
5. OpenAI. Function calling guide. OpenAI documentation.
6. Model Context Protocol. modelcontextprotocol.io.