AI crawler access check
Reads your robots.txt and tells you which AI crawlers are allowed in, agent by agent.
What this is actually checking
Check whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and nine other AI crawlers are allowed to fetch your site. Reads your real robots.txt. No signup.
An answer engine cannot cite a page it was never allowed to fetch. This is the first thing to rule out when a brand is absent from AI answers, and it is usually an accident rather than a decision.
The check distinguishes agents your file names explicitly from agents that only match a wildcard rule. Several crawlers read only their own block when one exists, so a file with a named group and a wildcard group can behave differently from how it reads.
The 2 checks behind it
- highAI crawlers are blocked
Blocking GPTBot, ClaudeBot, PerplexityBot and friends removes the site from AI answers.
- lowNo robots.txt
Without one, every crawler guesses, and you lose the place to declare your sitemap.
Does allowing AI crawlers mean my content trains a model?
It depends on the agent. Retrieval agents such as OAI-SearchBot, ClaudeBot, PerplexityBot and ChatGPT-User fetch a page to answer a question now, and they are the ones that decide whether you appear in an answer. Training agents such as CCBot gather corpora for later. You can allow the first group and refuse the second.
I allow them and still do not appear in answers. Why?
Access is necessary and not sufficient. The next two questions are whether the content exists in the served HTML before JavaScript runs, and whether any passage states a fact plainly enough to be quoted. Both are checked by the answer readiness tool.