AI Search Metric
LLM crawler share
The proportion of crawler traffic to your site coming from AI bots (GPTBot, ClaudeBot, Google-Extended, PerplexityBot) vs. classical crawlers (Googlebot, Bingbot). A leading indicator of upcoming citation activity.
What it is
LLM crawler share is the proportion of crawler traffic coming from AI bots such as GPTBot, ClaudeBot, Google-Extended, and PerplexityBot, relative to classical search crawlers. It shows how much of your crawl budget AI systems now consume.
Why it matters
It is a leading indicator of upcoming citation activity, because models tend to crawl and ingest content before they begin citing it, so rising AI crawl often precedes rising visibility.
How it works
Practitioners parse server logs by user agent to separate AI bots from traditional crawlers, then ensure those bots are allowed and can reach the content they want surfaced.
When it applies
Watch it as an early signal that AI systems are paying attention to your site before citations appear.
Examples
- GPTBot and ClaudeBot requests rising as a share of total crawls
- A site noticing AI crawl spikes shortly before new citations appear
- A team checking robots rules are not blocking the AI bots they want
- Comparing AI vs classical crawler share month over month
How it is measured
- AI bot requests divided by total crawler requests
- Crawl frequency per AI bot
- Pages crawled by AI bots vs classical crawlers
- Lead time between AI crawl increases and first citations
Related terms in AI Search Metric
- AI-referred trafficSessions referred from ChatGPT, Claude, Gemini, Perplexity, Copilot, and other LLM surfaces. Often under-counted in standard analytics because referrers are stripped or labelled as direct.
- Answer attribution rateThe proportion of AI answers that explicitly link back to source pages versus those that summarise without attribution. Attribution rate varies materially by platform and query type.
- Bounce-back rate (AI surface)The rate at which users return to the SERP or refine the prompt after seeing an AI Overview. Google's internal signal that an Overview failed, high bounce-back is associated with Overviews being silently removed for that query cluster.
- Brand mention rateThe frequency at which an LLM mentions a brand by name in answers across a prompt set, with or without citation. Brand mentions without citation are an early indicator of model-internalised brand awareness.
- Citation depthHow early in an AI-generated answer a source is cited. Earlier citations correlate with higher click-through from AI surfaces and stronger user trust signals.
- Citation diversity indexThe number of distinct domains an LLM cites across a prompt set, normalised by prompt count. Falling diversity in a category signals consolidation, and a closing window for new entrants.