Crawler access
Your homepage requested as each AI crawler, compared against a browser and Googlebot.
Free check, no signup
Can AI answer engines actually read your site?
Most sites that rank perfectly well on Google quietly return errors to the crawlers that fetch pages for ChatGPT, Claude and Perplexity. Nothing looks wrong in analytics, because Googlebot is still getting through. Enter a domain and find out in about ten seconds.
We request your homepage, your robots.txt and your llms.txt. Nothing else, and nothing is stored.
0
Crawler by crawler
| Crawler | HTTP | robots.txt | Verdict |
|---|
Email me this report
We will send the full report to your inbox so you can forward it to whoever owns the site.
What it checks
Access, not opinion.
robots.txt rules
Every crawler token evaluated properly, including conflicting groups a CDN may have prepended above yours.
Snippet suppression
Directives like nosnippet that stop engines quoting you even when the crawler reads the page fine.
Common questions
Straight answers.
What does this check actually test?
It requests your homepage using the user agent of each AI crawler, evaluates your robots.txt rules for each one, and looks for snippet suppression in your meta robots tag and X-Robots-Tag header. Every check is a deterministic read of public signals. No AI is used anywhere in it.
Why would my site block AI crawlers without me knowing?
Bot protection on CDNs often blocks AI crawlers by default, or behind a single toggle, while continuing to allow Googlebot. Search traffic looks normal, so nothing appears wrong in analytics. We found exactly this on a site we run.
Is a blocked result always correct?
No, and we say so in the report. Some CDNs verify crawlers by IP address as well as user agent, so a real crawler can be allowed where this test is refused. That is why the report compares against a browser and Googlebot, and tells you to confirm in your CDN settings before changing anything.
Does blocking training crawlers hurt me?
Much less than blocking citation crawlers. Refusing training use is a legitimate choice, and you can refuse it while still allowing the crawlers that fetch pages for live answers. It is the second group that decides whether you can be cited.
Fixed access is the start, not the finish.
Once crawlers can read you, the work is structure, entity consistency and the third-party pages engines actually cite. We run GEO and SEO as one program, on a site we engineer for both.
Book a 15-minute call →