What Is Crawlability?
Short answer
Crawlability is whether search engine and AI crawlers can discover and fetch your pages: reachable URLs, permissive robots rules, working links, and content that renders for bots. It's the first gate in visibility; nothing gets ranked, indexed, or cited that never gets crawled.
Crawlability explained
Crawling is discovery plus access. Crawlers find URLs through links and sitemaps, then fetch them subject to your permissions, robots.txt rules, bot-protection layers, and server behavior. Crawlability failures cluster in predictable places: orphan pages no internal link reaches, robots directives broader than intended, firewall and CDN rules blocking specific user agents, redirect chains and loops, and content that only materializes through JavaScript some bots never execute.
The concept is a budget as well as a gate. Large sites get crawled selectively, so wasted crawl, parameter duplicates, thin pages, infinite spaces, spends attention that should go to pages that matter, and planned architecture plus a clean XML sitemap direct it back. Small sites rarely have budget problems; they have access problems, usually inherited from a plugin or CDN default nobody reviewed.
AI multiplied the crawler population and the stakes. Training bots, retrieval bots, and now agents all consume the same crawl surface, and bot-protection vendors classify them with varying accuracy, which means sites are routinely uncrawlable to AI systems without anyone deciding that. The verification habit transfers unchanged from classic SEO: server logs are ground truth, showing exactly which agents fetched what, and any crawlability theory that contradicts the logs is wrong.
In practice
Crawlability is where I start every technical review, and the tooling makes it fast: a crawl of the site, a read of robots.txt and the CDN's bot rules, and a pass through server logs for who is actually fetching. The recurring finds are boringly consistent, orphaned money pages, a security setting bouncing AI crawlers, staging rules that shipped to production. None of it announces itself. The site looks fine to humans, which is exactly why the machines' view has to be checked directly.
Common misconception
People often assume anything published on the site is automatically crawlable. Discovery requires links or sitemaps and access requires permissive rules at every layer, and failures at any layer are invisible from a browser.
Free email course: Smarter SEO for the Age of AI
Seven short lessons over three weeks: why clicks are falling even when rankings hold, and how to get AI systems to recommend your brand.
