AI Voice Agent Latency: The Numbers Behind the Pause

Callers never time your phone agent with a stopwatch. They just feel it. Half a second of dead air after they finish a sentence and something reads as wrong, even when the words coming back are perfect. That pause is the main reason an AI voice agent gets clocked as a robot, and unlike "tone" or "personality" it is a number you can measure and fix.

Here is what the research and the vendor documentation actually say about that number.

Humans reply in about a fifth of a second

The most useful baseline comes from conversation research, not from telecom. Studying 10 languages across five continents, Stivers et al., PNAS 2009 measured the offset between the end of a question and the start of the answer. The mean response offset for the full dataset was +208 ms and the cross-linguistic median was +100 ms. Every language distribution peaked between 0 and 200 ms. The slowest language mean was Danish at +469 ms, the fastest Japanese at +7 ms.

That gives you a hard target. An AI voice agent that starts talking within roughly 200 to 500 ms of the caller going quiet sits inside the range humans already use with each other. Much past that and people fill the silence themselves. "Hello?" "Did that go through?" Then they talk over the agent, the agent restarts, and the call falls apart.

The phone line spends part of the budget before the model thinks

Voice quality standards treat delay on its own terms. ITU-T Recommendation G.114 (05/2003), One-way transmission time sets general limits for one-way mouth to ear delay: 0 to 150 ms is the preferred range where most speech applications feel essentially transparent, 150 to 400 ms is acceptable provided operators understand the quality impact, and above 400 ms is unacceptable for general network planning. The same recommendation notes that some highly interactive applications degrade even below 150 ms.

Read that carefully. G.114's 150 ms covers transport only, the time audio spends travelling. It says nothing about recognition or generation. Whatever the carrier, the codec and the jitter buffer consume, your agent gets the remainder.

Where the milliseconds go

A phone agent's response time is a chain, and every link adds to it:

• Endpointing. How long the system waits in silence before deciding the caller is done talking.
• Recognition finalization. Turning buffered audio into a final transcript, not just a partial one.
• Model time to first token. Long system prompts, big context and tool calls all push this out.
• Synthesis to first audio. Time until the first playable chunk, not the finished sentence.
• Transport both ways. Carrier hops, the media gateway, the jitter buffer.

Only the last one is what G.114 measures. The first four are design choices you control.

The silence timer is usually the biggest single chunk

This is where most teams quietly lose the call. Speech platforms wait for a fixed stretch of silence before they call the turn over. Microsoft documents the setting plainly: the segmentation silence timeout "can be set to integer values between 100 and 5000, in milliseconds, with 500 a typical default" (Microsoft Learn, How to recognize speech). The same page lists an initial silence timeout with typical defaults of 5000 ms for single shot recognition and 15000 ms for continuous recognition.

Sit with that first number. A 500 ms silence wait, by itself, is more than double the 208 ms mean gap humans use, and that is before a single token is generated or a single word is synthesized. Out of the box, a stock configuration is already late.

Newer realtime stacks attack this by predicting the end of a turn instead of counting silence. OpenAI's Realtime API documents two turn detection modes: `server_vad`, which "automatically chunks the audio based on periods of silence," and `semantic_vad`, which chunks when the model believes from the words themselves that the caller has finished, with an eagerness setting where "auto is the default value, and is equivalent to medium" (OpenAI, Voice activity detection). The broader guide points telephony agents at SIP as the transport and suggests starting with reasoning effort set to low for most production voice agents, then adjusting based on latency tolerance (OpenAI, Realtime API guide).

Buying the milliseconds back

Tune endpointing per question. A yes or no confirmation can end the turn after a short silence. An address or a phone number needs a longer one, because people pause in the middle of a string. One global timeout is a compromise that is wrong in both directions.

Keep the first response cheap. The opening line of a turn rarely needs deep reasoning. Acknowledge fast, then check the calendar while the caller is still hearing the acknowledgement.

Stream everything, and allow interruption. Play synthesized audio as the first chunk arrives rather than waiting for a complete sentence, and let the caller cut in at any point. An agent you can interrupt feels more human than one that is merely quick.

An illustrative budget, not a measurement from any specific deployment: 300 ms endpointing, 150 ms to a final transcript, 400 ms to first token, 200 ms to first audio and 100 ms of transport lands near 1.15 seconds. Roughly five times the human mean. Every one of those has to come down together, which is why "just use a faster model" seldom fixes a slow agent.

Where this doesn't apply

Speed is not the whole business case, and the Canadian evidence is sober about it. Statistics Canada reports that 19.2% of businesses used AI to produce goods or deliver services in the second quarter of 2026, up from 12.2% a year earlier, and that among AI-using businesses 28.2% reported using virtual agents or chatbots (Statistics Canada, June 11, 2026). But in a separate Statistics Canada study, Li and Liu found the association between AI adoption and firm productivity "falls to 5.1% and becomes statistically insignificant" once you control for complementary capabilities such as R&D, cloud computing and data analytics, concluding that adoption on its own is unlikely to deliver transformative gains (Economic and Social Reports, April 22, 2026). Tuning latency on a workflow nobody has fixed underneath is polish on the wrong surface.

Latency work also has sharp diminishing returns. Going from 1.2 seconds to 700 ms changes how a call feels. Going from 250 ms to 180 ms mostly does not, and the engineering to get there is not free.

Noise breaks endpointing. A contractor calling from a job site with a compressor running gets clipped mid sentence by aggressive turn detection, so you lengthen the timeout and give the latency back. Pick that tradeoff deliberately instead of discovering it in production.

And some calls should not be automated at all. Legal intake, medical triage and an angry customer belong with a person. The agent's job there is to route in seconds, not to hold the conversation.

If you record calls, the Office of the Privacy Commissioner of Canada is direct: an organization "must inform the customer that they are recording a call, clearly state the purpose of the recording and ask for their consent," and should offer an alternative to customers who object (OPC, Recording of Customer Telephone Calls). BC businesses have PIPA obligations on top of that. Put the disclosure in the greeting, where it costs one sentence.

What to actually measure

Log four timestamps on every turn: caller speech end as your endpointer sees it, final transcript, first token, first audio byte out. Then look at the median and the 95th percentile. The p95 is the one people remember, because that is the turn where the caller said "hello?" and everything after it went sideways. If you cannot produce those four numbers for last week's calls, you are not tuning latency. You are guessing.

Sources

• Stivers, T., Enfield, N. J., Brown, P., et al. "Universals and cultural variation in turn-taking in conversation." *Proceedings of the National Academy of Sciences*, 2009, 106(26), 10587 to 10592. https://pmc.ncbi.nlm.nih.gov/articles/PMC2705608/
• ITU-T. Recommendation G.114, "One-way transmission time," 05/2003. http://www.cs.columbia.edu/~andreaf/new/documents/other/T-REC-G.114-200305.pdf
• Microsoft. "How to recognize speech," Azure AI Speech documentation, Microsoft Learn. https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-recognize-speech
• OpenAI. "Voice activity detection (VAD)," Realtime API documentation. https://developers.openai.com/api/docs/guides/realtime-vad
• OpenAI. "Realtime API," developer documentation. https://developers.openai.com/api/docs/guides/realtime
• Statistics Canada. "Analysis on artificial intelligence use by businesses in Canada, second quarter of 2026," June 11, 2026. https://www150.statcan.gc.ca/n1/pub/11-621-m/11-621-m2026010-eng.htm
• Li, J. and Liu, H. "Artificial intelligence adoption and productivity in Canadian firms," Economic and Social Reports, Statistics Canada, April 22, 2026. https://www150.statcan.gc.ca/n1/pub/36-28-0001/2026004/article/00002-eng.htm
• Office of the Privacy Commissioner of Canada. "Recording of Customer Telephone Calls," updated March 6, 2018. https://www.priv.gc.ca/en/privacy-topics/surveillance/02_05_d_14/

Autana Solutions builds AI voice agents for shops, trades and clinics around Burnaby, New Westminster and the rest of Metro Vancouver, and latency tuning is part of the build rather than an upsell. If you want a straight read on whether your call volume justifies one, book a free call and we will go through a week of your phone log together.