TextGrain

What it is

Text grain describes how finely a body of writing is divided before it is processed by a search index, an embedding model or a retrieval-augmented generation pipeline. A coarse grain keeps whole pages or long chapters together, while a fine grain splits content into short passages, paragraphs or sentences. Where the term appears as a product or project name, the underlying idea is usually the same: working with text at a defined unit of analysis.

Why it matters

AI answer engines and retrieval systems rarely quote an entire page. They pull the passage that best matches the question, so the grain at which your content is written and structured affects whether it can be retrieved cleanly and attributed to you. Too coarse and the useful answer is buried in noise; too fine and the passage loses the context a model needs to use it correctly.

How it works

Practitioners set a grain when they chunk content for an index, choosing boundaries such as headings, paragraphs or fixed token windows, often with overlap so meaning is not cut in half. They then test retrieval quality at different grains, compare which passages get returned for real questions, and restructure source content so that each self contained answer sits inside one retrievable unit. Editorial teams apply the same thinking by writing short, standalone sections under clear headings.

When it applies

It applies whenever content is prepared for search indexing, embeddings, internal knowledge assistants or any retrieval-augmented generation system. It also applies in editorial planning, when deciding whether a topic should be one long page or several focused ones.

Examples

• A support team re-chunks its help centre from whole articles to one chunk per heading, so an assistant returns a single troubleshooting step rather than a 3,000 word guide.
• A B2B site rewrites a long pillar page so each subsection answers one question in full, making individual passages usable as standalone citations.
• An engineering team tests 200, 400 and 800 token chunk sizes with overlap and keeps the setting that returns the most relevant passage for a sample of real user questions.

How it is measured

• Retrieval precision and recall for a fixed set of test questions at each grain setting
• Share of retrieved passages that answer the question without needing neighbouring context
• Average chunk length and variance across the indexed corpus
• Citation rate of specific sections or passages in AI answers and assistant responses