Context window

What it is

The context window is the working memory of a language model for one exchange. It is counted in tokens rather than words, where a token is roughly a short word fragment, so a long brief, transcript or set of retrieved pages consumes the budget quickly. Different models expose different limits, and the limit covers both the input and the generated output.

Why it matters

Context limits decide how much of your content an AI assistant can actually hold while answering a question, which affects whether your page is used in full or reduced to a fragment. In AI search and retrieval systems, a crowded window means only the highest-scoring chunks make it through, so structure and self-contained passages matter more than page length. For teams building assistants, the window also drives cost and latency, because every token in the window is paid for and processed.

How it works

Practitioners measure token counts before sending requests, then manage the budget by trimming system prompts, summarising earlier turns, and retrieving a small number of relevant chunks rather than whole documents. Long tasks are broken into stages, with intermediate summaries carried forward instead of raw text. Content teams help by writing sections that make sense on their own, with clear headings and definitions near the top, so a single retrieved chunk still answers the question.

When it applies

It applies whenever you prompt a model with long inputs, build retrieval augmented generation, run document analysis, or maintain multi-turn chat where the history keeps growing.

Examples

• A support assistant loses details from the start of a long chat because earlier turns were dropped to make room for new messages and retrieved help articles.
• An analyst pastes a 90-page annual report into a model and gets vague answers, then splits it into chapters with per-chapter summaries to stay inside the limit.
• A retrieval pipeline is capped at eight passages of around 400 tokens each so the prompt, sources and answer all fit within the model's limit.

How it is measured

• Average and peak tokens per request, shown as a percentage of the model's limit
• Truncation rate: share of requests where history or retrieved content had to be dropped
• Number and size of retrieved chunks passed into each prompt
• Cost and latency per request, tracked against token volume