Algorithm
Search index
Also known as: search engine index, indexing
A search index is the structured store of web content a search engine builds from crawling and processing pages, used to look up and rank results at query time. It typically maps words, entities and features back to the documents that contain them, along with signals about each document. If a page is not in the index, it cannot appear in results drawn from that index.
What it is
A search index is the database a search engine queries when someone searches, rather than the live web itself. It is usually built as an inverted index, where terms and entities point to lists of documents, supported by stored metadata such as canonical URL, language, freshness and quality signals. Modern indexes also store vector or embedding representations so systems can retrieve by meaning as well as by exact words.
Why it matters
Indexing is the gate between publishing and being found: crawling, rendering and indexing all have to succeed before ranking is even possible. Many AI answer systems and assistants retrieve from a search index or a similar retrieval layer, so indexing problems now suppress visibility in AI surfaces as well as classic blue links. Large sites often lose traffic not because content is weak but because thin, duplicate or parameterised URLs consume crawl and indexing attention.
How it works
Search engines discover URLs from links, sitemaps and feeds, fetch them, render where needed, canonicalise duplicates, extract text and structured data, then write the result to the index. Practitioners influence this with clean internal linking, XML sitemaps, canonical tags, robots directives, fast responses and by removing low value URLs. Diagnosis usually runs through Google Search Console Page Indexing reports, Bing Webmaster Tools, server log analysis and site: style spot checks.
When it applies
Indexing matters continuously, but it becomes urgent during site migrations, replatforming, large content launches, and on any site with faceted navigation or user generated pages.
Examples
- An ecommerce site finds 40,000 filter URLs marked as crawled but not indexed, and blocks the parameter combinations while keeping key category pages canonical.
- A publisher submits a news sitemap and sees new articles indexed within minutes, while older archive pages take weeks.
- A B2B site relaunches, forgets to remove the staging noindex tag, and loses all organic visibility until the tag is stripped and URLs are resubmitted.
How it is measured
- Indexed pages versus submitted pages in Google Search Console Page Indexing
- Ratio of indexed URLs to total valuable URLs, sometimes called index coverage or index bloat
- Time from publish to first impression for new URLs
- Crawl requests and average response time from crawl stats or server logs
Insights on Search index
Related terms in Algorithm
- Core updateA broad Google algorithm update affecting search rankings sitewide rather than a single feature. Recent core updates (e.g. May 2026) have increasingly favoured intent-aligned content over keyword-optimised content.
- EEATGoogle's quality framework: Experience, Expertise, Authoritativeness, Trustworthiness. EEAT signals are increasingly weighted in both classical ranking and AI Overview source selection.
- Entity salienceEntity salience is a measure of how central an entity is to a piece of content, as opposed to merely being mentioned in it. Natural language processing systems assign a salience score based on signals such as position, frequency, grammatical role and surrounding context. It helps a system decide what a document is actually about.
- Google Search algorithmThe Google Search algorithm is the collection of ranking and retrieval systems Google uses to find, order and present results for a query. It is not a single formula but many interacting systems that assess content, links, query intent, context and user signals. Google updates these systems continuously, with larger changes announced as core updates.
- Google Spam UpdateA Google Spam Update is a rollout of changes to the systems Google uses to detect and demote content that breaks its spam policies. These updates target things like scaled content abuse, cloaking, hacked pages, link schemes and site reputation abuse rather than general content quality. Sites hit by a spam update usually need to remove the offending practices before they can recover.
- Helpful Content UpdateGoogle's quality-focused update class targeting low-value, AI-generated, or thin content. Successive HCUs have favoured first-hand experience and named expertise over comprehensive but shallow coverage.
Primary research · August 2026
How ChatGPT Shortlists Software Brands
An audit across 10 categories and 60 buying questions. I recorded what ChatGPT reads, throws away and links to when a buyer asks it which software to buy, and what that decides.
- 60
- Questions asked
- 10
- Software markets
- 2,680
- Results read
- 367
- Links shown
Free download
Get the full report
35 pages · PDF · 536 KB. Enter your details and it downloads straight away.