What Are LLM Crawlers?
Short answer
LLM crawlers are the bots AI companies send to read websites, such as OpenAI's GPTBot and OAI-SearchBot and Anthropic's ClaudeBot. Some collect training data and some fetch pages to answer questions live. If they can't reach your site, AI systems can't learn from it or cite it.
LLM Crawlers explained
Every AI company that reads the web identifies its bots with named user agents, and most now run separate bots for separate jobs. Knowing which is which lets you decide what to allow in robots.txt.
Three jobs, three kinds of bot
- Training crawlers collect content that may be used to train future models. Blocking them affects what models learn about you over time.
- Search crawlers build the index an AI search product draws on. Blocking them removes you from that product's cited answers.
- User-triggered fetchers visit a page when a person asks the assistant to. They don't crawl on their own.
The main AI crawlers
- OpenAI (documentation): GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (user requests; OpenAI says robots.txt rules may not apply to it).
- Anthropic (documentation): ClaudeBot (training), Claude-SearchBot (search), Claude-User (user requests). Anthropic says its bots follow robots.txt.
- Google (documentation): Google-Extended is a robots.txt token, not a separate bot. It controls whether content Google crawls can be used for Gemini training and grounding. Google says it doesn't affect inclusion or ranking in Google Search.
- Perplexity: PerplexityBot (search index) and Perplexity-User (user requests).
- Common Crawl: CCBot, whose open web dataset many AI models are trained on.
Google's AI Overviews work differently
AI Overviews and AI Mode use regular Google Search crawling. Google's AI features documentation says a page only needs to be indexed and eligible to show a snippet. Blocking Google-Extended won't take you out of them.
Access comes first
Security plugins, firewalls and CDN bot settings often block AI user agents by default, and those settings change without anyone noticing. To check access:
- Read your robots.txt for rules naming the bots above.
- Review your CDN or firewall bot management settings.
- Search your server logs for these user agents. If they never appear, that's your finding.
In practice
I run a five-minute access check before any AI visibility work, because I keep finding forgotten crawler rules: a security plugin update, a CDN bot-score threshold, an old blanket bot block. In one case GPTBot and ClaudeBot had been blocked for months. The client saw no error, just no citations.
Common misconception
If Googlebot can crawl your site, it's easy to assume AI crawlers can too. They're separate user agents with separate rules, and plenty of sites ended up blocking or allowing them by accident.
Sources
Free email course: Smarter SEO for the Age of AI
Seven short lessons over three weeks: why clicks are falling even when rankings hold, and how to get AI systems to recommend your brand.
