Crawlability
TL;DR: What is Crawlability?
Crawlability is whether search engine and AI crawlers can discover and fetch your pages: reachable URLs, permissive robots rules, working links, and content that renders for bots. It is the first gate in visibility; nothing gets ranked, indexed, or cited that never gets crawled.
Crawlability explained
Crawling is discovery plus access. Crawlers find URLs through links and sitemaps, then fetch them subject to your permissions, robots.txt rules, bot-protection layers, and server behavior. Crawlability failures cluster in predictable places: orphan pages no internal link reaches, robots directives broader than intended, firewall and CDN rules blocking specific user agents, redirect chains and loops, and content that only materializes through JavaScript some bots never execute.
The concept is a budget as well as a gate. Large sites get crawled selectively, so wasted crawl, parameter duplicates, thin pages, infinite spaces, spends attention that should go to pages that matter, and deliberate architecture plus a clean XML sitemap direct it back. Small sites rarely have budget problems; they have access problems, usually inherited from a plugin or CDN default nobody reviewed.
AI multiplied the crawler population and the stakes. Training bots, retrieval bots, and now agents all consume the same crawl surface, and bot-protection vendors classify them with varying accuracy, which means sites are routinely uncrawlable to AI systems without anyone deciding that. The verification habit transfers unchanged from classic SEO: server logs are ground truth, showing exactly which agents fetched what, and any crawlability theory that contradicts the logs is wrong.
In practice
Crawlability is where I start every technical review, and the tooling makes it fast: a crawl of the site, a read of robots.txt and the CDN's bot rules, and a pass through server logs for who is actually fetching. The recurring finds are boringly consistent, orphaned money pages, a security setting quietly bouncing AI crawlers, staging rules that shipped to production. None of it announces itself. The site looks fine to humans, which is exactly why the machines' view has to be checked directly.
Common misconception
People often assume anything published on the site is automatically crawlable. Actually, discovery requires links or sitemaps and access requires permissive rules at every layer, and failures at any layer are invisible from a browser.
Want the bigger picture? Start with the Founder’s Guide to SEO.