We measured 100 SaaS sites for AI crawler access.

The sample, stated before the conclusions

A convenience sample of 100 well-known SaaS, developer-tool and marketing-tool companies. Not a random sample and not a ranking. These are companies whose marketing sites are public, well-resourced and widely imitated, which makes them the interesting case: if sites like these get AI crawler access wrong, the long tail is worse rather than better.

Of 100 domains, 90 answered and 10 did not. Every percentage on this page divides by 90, and that is worth repeating because a rate over an unstated denominator is the single most common way a study of this kind misleads.

Method

• Two requests per domain, both public: robots.txt and the homepage. Nothing else was crawled, nothing was stored about anyone, and there was a delay between sites.
• Crawler access is read from robots.txt for the six retrieval agents that decide whether a brand can appear in an AI answer today: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended. Training-only crawlers are excluded, because refusing those is a defensible business decision while refusing these makes you invisible.
• Everything else is read from the HTML the server returned, before any JavaScript ran, because that is what a retrieval crawler receives.
• An llms.txt counts only when the response is 200, served as text/plain, and begins with a Markdown heading. A first pass without those conditions reported 70%; the real figure is 57.8%.

The fourth point is the one worth dwelling on. A first pass of this study reported that 70% of the sample published an llms.txt, which would have been a striking headline and a wrong one: plenty of sites answer 200 with an HTML shell at any path, and one answered 403 to a browser and 200 to this client within the same minute. Requiring a text/plain response and a Markdown heading moved the real figure to 57.8%. A false positive is more expensive than a miss, and in published research it is fatal.

Who actually blocks a retrieval crawler

3 sites in the sample disallow at least one of the six agents from the root. The pattern is consistent: companies whose product is the content or the canvas itself.

The homepages a crawler sees as almost blank

None of the major AI retrieval crawlers executes JavaScript. These are the word counts in the HTML the server actually returned, which is the whole of what those crawlers receive. The median across the sample is 910 words.

A low number here is not proof that a company is invisible in AI answers. It means the homepage carries almost nothing an answer engine can quote, and that whatever authority the brand has is being earned somewhere other than its own front page. The measurement behind the JavaScript claim is a separate page, with the sources.

The sites that refused

10 of 100 answered 403, 429 or 400 to a plain, identified, rate-limited request at a delay of well under one request per second. They are listed because excluding them silently would change every percentage on this page without saying so.

Every measurement, per site

All 100 domains, in the order they were read. Anyone can reproduce a row with two requests.

Questions