LLM inference

What it is

Inference is the serving side of a language model: weights are already fixed, and the system spends compute turning an input prompt into an output sequence. It usually runs in two phases, a prefill phase that reads the prompt and a decode phase that generates tokens one after another. Anything the model is given at that moment, including retrieved web pages or documents, forms the context it reasons over.

Why it matters

AI answers about your brand are produced at inference time, which means the content pulled into the context window has a direct effect on what gets said. Inference also costs money and time per token, so assistants have practical limits on how many sources they read and how much of each page they keep. Understanding those limits explains why concise, well structured pages are quoted more often than long, unstructured ones.

How it works

Practitioners rarely run inference themselves, but they work with its constraints: keeping key facts high on the page, using clear headings and short passages that survive chunking, and making sure the retrieval layer can find a clean answer quickly. Engineering teams tune it with batching, caching of repeated prompts, quantised weights and smaller models for routine tasks. Where a business runs its own assistant, inference cost and latency become a budget line that shapes product decisions.

When it applies

It applies whenever a live model is generating an answer, including AI search results, chat assistants, support bots and content tools built on an API.

Examples

• A shopper asks an assistant to compare two broadband packages, and the model performs inference over retrieved pricing pages before writing a summary.
• An in-house support bot caches common prompts so repeat questions skip part of the inference cost.
• A content team notices answers get truncated for a long buying guide, and splits it into shorter pages so more of the useful text fits into the context passed at inference.

How it is measured

• Time to first token and total response latency for assistant answers
• Tokens consumed per request, input and output, as a cost driver
• Share of retrieved pages that actually appear in the generated answer
• Error and truncation rate for long or complex prompts