robots.txt
TL;DR: What is robots.txt?
Robots.txt is a plain text file at the root of your website that tells crawlers which URLs they may access. It now also governs AI crawlers like GPTBot and ClaudeBot, which makes it the first control lever for your AI visibility.
robots.txt explained
Robots.txt implements the Robots Exclusion Protocol, a convention dating to 1994 that lets site owners give crawling instructions to bots. The file lives at yourdomain.com/robots.txt and uses simple user-agent and disallow rules to state which parts of a site each crawler may fetch.
Two mechanics trip people up. First, robots.txt controls crawling, not indexing: Google's own documentation notes that a blocked URL can still appear in search results if other pages link to it. Removing a page from search requires noindex or removal, not a disallow rule. Second, the file is public and advisory, so it is never a security mechanism.
The stakes changed when AI crawlers arrived. OpenAI introduced GPTBot in August 2023, and Anthropic, Perplexity, and Google (via Google-Extended) followed with their own user agents. Your robots.txt now decides whether AI systems can read your content at all. Block those crawlers and you may protect content from training, but you also remove yourself from the answers your buyers are reading.
One nuance worth knowing: Google-Extended governs whether your content trains Gemini models, but it does not remove you from AI Overviews, which are fed by ordinary Google Search crawling. Controlling one does not control the other, and mixing them up is one of the most common AI crawler mistakes.
In practice
In audits I keep finding AI crawler rules nobody remembers adding. Security plugins and CDN bot-protection settings quietly block GPTBot or ClaudeBot by default, and the site owner has no idea they are invisible to AI search. Before any AI visibility work, I pull the robots.txt and the CDN bot rules first. Five minutes there can explain months of missing citations.
Common misconception
People often think robots.txt hides pages from Google. Actually, disallowed pages can still be indexed through links from other sites. If you need a page out of search results, use noindex; if you need it private, use authentication.
Related terms
Want the bigger picture? Start with the Founder’s Guide to SEO.