Web crawling

What it is

A crawler starts from a set of known URLs, requests each page, extracts links and queues new URLs to visit, subject to rules about scope, depth and rate. It reads robots.txt, respects or ignores directives depending on the operator, and records response codes, redirects and content. Modern crawlers may also render JavaScript to see content that only appears after the page loads.

Why it matters

If a page is never crawled it cannot be indexed, cited or retrieved, so crawlability is the entry condition for organic and AI visibility. Crawl behaviour also reveals technical problems such as broken links, redirect chains, orphan pages and wasted budget on faceted URLs. With AI crawlers now fetching content for training and for live retrieval, decisions about what to allow or block have commercial consequences.

How it works

Site owners manage crawling with robots.txt rules, XML sitemaps, internal linking, canonical tags, response codes and server performance. Practitioners run their own crawls with tools such as Screaming Frog or Sitebulb to audit a site, and read server log files to see what real search and AI crawlers actually fetched. Slow responses, heavy JavaScript and thin duplicate URLs all reduce how much of a site gets crawled.

When it applies

It applies to every site that wants to be found, and especially after migrations, redesigns, large content launches or changes to URL structure. It also applies whenever a business is deciding whether to allow or block specific AI user agents.

Examples

• An ecommerce team blocks crawling of filtered listing URLs so crawlers spend their time on category and product pages.
• A publisher reviews server logs and finds most crawl requests hitting old paginated archives rather than new articles.
• A B2B site adds internal links from its main resources hub to orphaned case studies so crawlers can reach them.

How it is measured

• Crawl requests per day by user agent from server logs
• Ratio of crawled URLs to indexed URLs
• Share of crawl requests returning non-200 status codes
• Average server response time for crawler requests