LLM Crawlers
TL;DR: What is LLM Crawlers?
LLM crawlers are the bots AI companies use to read the web: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, and others. They gather content for model training and live retrieval, and whether they can access your site is the floor under all AI visibility.
LLM Crawlers explained
The fleet arrived quickly. OpenAI introduced GPTBot in August 2023, and Anthropic, Perplexity, Common Crawl's CCBot, and Google's Google-Extended token established the pattern: each AI company operates named user agents, controllable through robots.txt like any crawler.
The crawlers do different jobs, and the distinctions matter. Training crawlers gather content for future model knowledge; blocking them keeps you out of what models learn. Retrieval and search bots (OAI-SearchBot, ChatGPT-User, PerplexityBot in its live mode) fetch pages to answer user queries right now; blocking them removes you from cited answers today. And Google's split is the most misunderstood: Google-Extended controls Gemini training only, while AI Overviews run on ordinary Googlebot crawling, so blocking one does not touch the other.
Access, not optimization, is the first battle. Bot-protection layers in security plugins and CDNs ship with default rules that silently block AI user agents, and firewall categories update without site owners noticing. The diagnostic is straightforward: read your robots.txt, review your CDN's bot management settings, and check server logs for the named agents actually fetching pages. A site invisible to LLM crawlers is invisible to LLM answers, no matter how well the content is built. Allow decisions can be selective, but they should at least be deliberate.
In practice
This is the five-minute check I run before any AI visibility work, because I keep finding crawler rules nobody remembers adding. A security plugin update, a CDN bot-score threshold, a blanket bot block from an old spam problem, and suddenly GPTBot and ClaudeBot have been bouncing off the site for months. The client sees no error; they just see no citations. Server logs settle it fast: if the AI user agents are not in your logs, that is the finding, and everything else waits.
Common misconception
People often assume that if Google can crawl a site, AI systems can too. Actually, AI crawlers are separate user agents governed by separate rules, and blocking or allowing them is a set of independent decisions many sites made by accident.
Related terms
Want the bigger picture? Start with the Founder’s Guide to SEO.