AI crawler
An AI crawler is an automated fetcher operated by a model provider that reads web pages either to answer a user's question in real time or to build training data.
The two purposes behave differently and are worth separating. Retrieval agents such as OAI-SearchBot, ClaudeBot, PerplexityBot and ChatGPT-User fetch a page because someone asked a question now, and they decide whether a brand appears in that answer. Training agents such as CCBot gather corpora for later.
A site can allow one and refuse the other, and robots.txt is where that decision is expressed. Blocking retrieval agents is the most direct way to become invisible in AI answers, and it is usually done by accident rather than on purpose.
Related
Tell us about your business.
Thirty minutes with a specialist. You leave with a plan, whether or not you hire us.