robots.txt generator
Builds a robots.txt that names each search and AI agent, with your sitemap included.
What this is actually checking
Generate a robots.txt that explicitly allows or refuses each AI crawler by name, rather than leaving it to a wildcard rule, with your real sitemap URL filled in.
Naming each agent rather than relying on a wildcard removes the ambiguity that causes most accidental blocks, because several crawlers read only their own group when one exists.
The generated file separates retrieval agents, which decide whether you appear in an answer today, from training agents, which build corpora for later. Those are different decisions and a single wildcard cannot express both.
The 3 checks behind it
- lowNo robots.txt
Without one, every crawler guesses, and you lose the place to declare your sitemap.
- highAI crawlers are blocked
Blocking GPTBot, ClaudeBot, PerplexityBot and friends removes the site from AI answers.
- criticalrobots.txt blocks a page that should rank
Disallowed URLs cannot be crawled, so their content never enters the index.
Should I block AI training crawlers?
It is a business decision rather than an SEO one. Blocking training does not remove you from answers that retrieval agents produce, and blocking retrieval does remove you. If in doubt, allow retrieval and decide about training separately.