What Is robots.txt?
Short answer
Robots.txt is a plain text file at the root of a website that tells crawlers which URLs they may fetch. It also controls AI crawlers like GPTBot, OAI-SearchBot and ClaudeBot, so it decides whether AI systems can read your site at all.
robots.txt explained
Robots.txt follows the Robots Exclusion Protocol, a convention from 1994 that became an internet standard, RFC 9309, in 2022. The file lives at yourdomain.com/robots.txt. Each group of rules names a crawler with User-agent, then lists the paths it may not fetch with Disallow (or may fetch, with Allow).
Two things robots.txt doesn't do
- It doesn't remove pages from Google. It controls crawling, not indexability. Google's robots.txt documentation notes that a blocked URL can still appear in results if other pages link to it. Use noindex to keep a page out of search.
- It isn't security. The file is public and following it's voluntary. Anything private needs a login.
The AI crawlers you're now deciding about
Most AI companies run separate bots for model training and for search, and you can treat them differently:
- OpenAI: GPTBot collects training data. OAI-SearchBot decides what can appear in ChatGPT search. ChatGPT-User fetches a page when a person asks ChatGPT to, and OpenAI's crawler documentation says robots.txt rules may not apply to it.
- Anthropic: ClaudeBot collects training data, Claude-SearchBot indexes for search, and Claude-User fetches pages for a person's question. Anthropic's help article says all three follow robots.txt.
- Google: Google-Extended controls whether your content is used to train Gemini.
- Perplexity: PerplexityBot indexes pages for Perplexity's answers.
Here's a file that keeps a site open to AI search while opting out of training:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /Which setup is right depends on the business. A publisher that sells its content may block training. A B2B company that wants buyers to hear its name in AI answers usually wants all of these bots in.
Check yours in five minutes
- Open yourdomain.com/robots.txt. Look for groups naming the bots above, or a
User-agent: *group withDisallow: /. - Check your CDN and security plugins. Some bot-protection settings block AI crawlers before robots.txt is ever read.
- Search your server logs for these user agents to confirm they're reaching your pages.
In practice
In audits I keep finding AI crawler blocks nobody remembers adding. Security plugins and CDN bot-protection settings can block GPTBot or ClaudeBot by default, and the site owner has no idea they're invisible to AI search. Before any AI visibility work, I check robots.txt and the CDN bot rules first. Five minutes there can explain months of missing citations.
Common misconception
Many site owners assume blocking Google-Extended takes them out of AI Overviews. It doesn't. AI Overviews come from regular Google Search crawling and indexing. Google-Extended only controls whether your content is used for Gemini training and grounding in Google's other AI products.
Sources
Free email course: Smarter SEO for the Age of AI
Seven short lessons over three weeks: why clicks are falling even when rankings hold, and how to get AI systems to recommend your brand.
