A web app that waits on a slow API shows a spinner. A voice agent that waits on a slow API plays dead air. The caller can't see a loading state. All they hear is that the line went quiet, and on a phone call, quiet means something.
Every demo of agentic voice I've seen runs on the happy path. The caller asks where their order is, the agent calls the order system, the order system answers in 300 milliseconds, the agent reads back the status. Nobody demos the version where the CRM takes four seconds, or times out, or half-completes a write. That's the version your callers will hit on a busy Monday morning. So that's the version worth engineering.
The budget comes from humans, not from your stack
Conversation runs on a tight clock, and the research behind that is solid. Stivers and colleagues, writing in PNAS in 2009, measured turn-taking across 10 languages, from major world languages to small indigenous communities. Every one of them showed the same two things: speakers avoid overlapping each other, and they keep the silence between turns as short as possible. The languages did differ in average gap length, but only within 250 ms of the cross-language mean. Humans hand off turns in fractions of a second. Callers carry that expectation into every call, and they don't drop it because a machine picked up.
Telecom engineers have known this for a long time. ITU-T Recommendation G.114 is the one that tells network planners how much one-way transmission delay a voice path can carry, and it was written for the network alone. That's codecs, packetization, jitter buffers and transport. No speech recognition, no language model, no text-to-speech. Our industry spent decades shaving milliseconds off that path. Then agentic voice showed up and stacked endpoint detection, model inference, speech synthesis and a round trip to someone else's SaaS API on top of it.
That's the real constraint. Tool latency isn't a backend metric. It comes straight out of the caller's patience.
The protocol gives you the mechanism, not the number
The Model Context Protocol is honest about this. The current lifecycle spec says implementations should set timeouts on every request they send. When a timeout fires, the sender should issue a cancellation and stop waiting. Progress notifications can reset the clock, but implementations should still enforce a maximum timeout no matter what progress gets reported, so one misbehaving server can't hang the session.
That's the right design, and it's also where the spec stops. MCP can't tell you what the timeout should be, because the right answer depends on who's waiting. A batch job can wait thirty seconds. A caller on hold with an AI agent can't. You have to set the number per tool, and you have to set it on purpose.
Three classes of tool call, three failure contracts
In practice I sort mid-call tool calls into three classes and design each one differently.
Fast reads are things like order status, store hours and appointment availability. The agent should be able to answer these inside a normal conversational turn. If one blows its budget, the agent should say so plainly and offer a path forward. It shouldn't quietly retry three times while the caller wonders if the call dropped.
Slow reads are things like complex account lookups or multi-system joins. The agent can cover these with honest speech: "Let me pull that up." But filler is a promise. If the agent says it's checking, it owes the caller an answer or an explanation within a few seconds. Filler that runs on and on is just dead air with better production values.
Writes are bookings, order changes and payment-adjacent actions, and this is where most implementations get it wrong. A write that timed out hasn't necessarily failed. The appointment may well exist in the scheduling system even though the response never came back. If the agent tells the caller "I wasn't able to book that" and then retries, you've built a double-booking machine. Writes need idempotency keys, a reconcile-before-retry step, and agent language that can carry uncertainty: "I've sent the booking. You'll get a confirmation text within a few minutes, and if you don't, here's what happens next."
The fallback for all three should be the same: capture what the caller needed, and either schedule a callback or hand them to a person with the context attached. The worst outcome isn't a failed lookup. It's a caller who hangs up without knowing whether anything happened.
Instrument the silence
You can't tune a budget you don't measure. For every tool call, you want the connector, the duration, the outcome (success, error, timeout or cancelled), and, critically, what happened to the call next. Did the caller stay? Did they get escalated? Did they hang up within ten seconds of a timeout?
That last join is where most standalone voice-agent stacks fall apart, because the tool logs, the call record and the outcome sit in three different systems. It's also why I care that Dial800 AI Voice Agents log every interaction with timestamps, actions taken, routing decisions and outcomes, in the same call record that carries the recording, the transcript and the source attribution. The tool calls run through a server-side MCP gateway with per-tool allow-listing. So the question "which connector is costing us callers?" is a filter on data you already have, not a cross-system forensics project. The same goes for the escalation side. When the AI hands a caller to your contact-center queues, the context summary goes with them, so a timed-out lookup doesn't turn into the caller repeating everything from the beginning.
To be clear about what's ours to own and what isn't: Dial800 can't make your CRM faster. Neither can any voice platform. What a platform can do is make slowness visible, cap how long a caller waits on it, and fail toward a human instead of toward silence.
The question to ask
Next time you evaluate a voice agent, skip the happy-path demo. Ask what happens when the backend takes four seconds. Then ask what happens when a booking request times out after the booking was actually written. If the answer is a shrug, you're not looking at an agent yet. You're looking at a demo.