Latency

What it is

Latency covers every delay in the chain: network time, queueing, model inference, tool calls and any live web fetches. Time to first token measures perceived responsiveness, while tokens per second and total time measure how quickly the full answer arrives. In retrieval heavy systems, fetching and reading source pages often contributes more delay than the model itself.

Why it matters

Slow answers get abandoned, and slow sources get dropped. When an assistant fetches pages at query time, it usually works to a timeout, so a page that responds slowly may be skipped in favour of a faster one, which costs you a citation. Latency also shapes product design, because a tight budget forces fewer retrieved documents and shallower reasoning.

How it works

Practitioners instrument each stage separately, then look at the distribution rather than the average, because tail latency drives complaints. They reduce it by streaming output, caching frequent requests, retrieving fewer but better documents, using smaller models for simple steps and running work in parallel. On the publishing side, they cut server response time, keep content in the initial HTML and avoid redirect chains so crawlers and live fetchers can read pages quickly.

When it applies

It applies to any live AI feature you operate, and to your own site whenever assistants and crawlers fetch your pages at query time.

Examples

• A support assistant streams its first sentence within a second while continuing to retrieve documents in the background.
• A publisher cuts time to first byte on article pages so live fetching agents do not time out before reading the content.
• An agent workflow runs three retrieval calls in parallel rather than in sequence, halving total answer time.

How it is measured

• Time to first token, at median and 95th percentile
• Total answer completion time and tokens per second
• Server time to first byte and full render time for your key pages
• Abandonment rate, measured as sessions where the user leaves before the answer finishes