What Is llms.txt?
Short answer
llms.txt is a proposed plain-text file at a site's root listing its most important content in a format built for AI systems to read. Proposed by Jeremy Howard in 2024, it remains unratified, largely unread by AI search crawlers, and most useful for AI agents and developer docs.
llms.txt explained
Jeremy Howard of Answer.AI proposed llms.txt in September 2024 as a Markdown index helping language models find a site's essential content without parsing full HTML. The idea spread fast; the evidence has been sobering. An SE Ranking study of 300,000 domains found 10.13% adoption and no statistically significant correlation with AI citations. An Ahrefs analysis of 137,000 sites in May 2026 found 97% of published llms.txt files received zero requests; AI search crawlers like GPTBot and PerplexityBot overwhelmingly crawl HTML directly.
Google has been unusually direct. Its June 2026 documentation states plainly that Search, including its AI features, ignores the file, and John Mueller compared it to the discredited keywords meta tag: a self-declared description machines can't trust precisely because every site would declare itself excellent.
The honest current picture splits by audience. For AI search visibility, llms.txt does nothing measurable. For AI agents and coding assistants, it's already in real use: tools like Cursor and Claude Code fetch it for clean documentation, companies like Stripe, Vercel, Cloudflare, and Anthropic ship it for exactly that audience, and Chrome's Lighthouse now audits it under an Agentic Browsing category. The file's future, if it has one, is agentic infrastructure, not a citation lever.
In practice
Early on, a lot of us were rightfully hesitant here, and the data has backed that caution: no confirmed use by the major LLMs for search, no citation correlation. My counsel to clients is still pragmatic. It's lightweight, there's no downside, and the agentic web may give it a real job yet. And there's a hidden benefit nobody markets: writing one forces you to describe your brand and your content succinctly and clearly, which is a worthwhile exercise even if no bot ever reads the file.
Common misconception
People often implement llms.txt expecting AI search citations to improve. The effect on citations so far is zero; its real, growing use case is AI agents and coding assistants fetching clean documentation on demand.
Sources
Free email course: Smarter SEO for the Age of AI
Seven short lessons over three weeks: why clicks are falling even when rankings hold, and how to get AI systems to recommend your brand.
