How to Check Whether Competitors Block AI Crawlers

A competitor's crawl posture is one of the few strategic facts about them that is genuinely public. Every site publishes robots.txt at a fixed path for crawlers to read, so checking which AI crawlers an incumbent allows costs one HTTP request per domain and the time it takes to read one small text file. How long a whole market takes is a function of how many incumbents you list, and we are not going to quote you a number we have not timed.

This page is the method, its single legitimate inference, and the several inferences it cannot support. It reads other people's files; deciding your own posture is a different question, answered in should publishers block AI crawlers, and auditing your own brand's visibility is a different exercise again.

Which product tokens should you look for?

The 11 tokens with a retrieval or training stake, copied verbatim from the platforms' own crawler documentation. Guessing at a token name is the most common way this audit produces a wrong table — a misspelled token matches nothing, and a token nobody publishes does not exist.

Three entries in that table behave unlike the rest. Google-Extended has no user agent string of its own, so no request will ever arrive under that name — it is a policy line only, and Google documents that it does not affect Search inclusion. And the -User tokens are the protocol's gray zone: OpenAI states robots.txt rules "may not apply" to ChatGPT-User and Perplexity states Perplexity-User "generally ignores" them, while Anthropic documents Claude-User as respecting them. A disallow on those two lines is a request, not a mechanism, and your table should say so.

This checklist is the crawl-posture subset, not the full census — the complete inventory of who fetches what, including tokens outside these 11, is in the complete AI crawler list.

How does robots.txt matching actually work?

A crawler picks the group whose user-agent line matches the product token it declares, and falls back to the wildcard group when no specific group matches. RFC 9309 standardizes that format and the matching rules, including case-insensitive token matching and the most-specific-group-wins behavior.

Three parsing details decide whether your table is right:

• A specific group replaces the wildcard group; it does not add to it. A site with a permissive User-agent: * group and a restrictive User-agent: GPTBot group is blocking GPTBot completely, and a site with a restrictive wildcard group and no GPTBot group is blocking GPTBot too — via the wildcard.
• Path rules are prefix-based. A disallow on a section is not a site-wide block, so record "partly disallowed, /section/" rather than flattening it to a yes or no.
• The file is a declaration, not an enforcement mechanism. RFC 9309 defines the format and how conforming crawlers match it; whether a given client obeys it is that client's choice, which each platform states in its own crawler documentation.

Record the fetch date next to every row. Robots.txt is edited without announcement, and a table without a date is an assertion about a moment nobody can identify.

What can you legitimately conclude?

Eligibility, and nothing beyond it. A site that disallows a search-index crawler is not in that engine's retrieval pool for as long as the rule stands — that is the vendors' own documented behavior for their own crawlers, and it is a fact about mechanics rather than a prediction about anyone.

The distinction in the second row is where most competitor analyses fail. OpenAI documents GPTBot and OAI-SearchBot as separate tokens for separate jobs, and Google documents Google-Extended as governing Gemini training and grounding rather than Search. A site can decline training and remain fully retrievable, so "they block AI" is almost always an imprecise reading of a file that says something narrower. The decision logic behind those distinctions is laid out in the robots.txt decision framework for AI.

What must you never conclude?

Motive, and consequences. A robots.txt file records a rule, not a reason: a disallow can come from a licensing negotiation, a legal policy, a server-load decision, a CDN default nobody reviewed, or a contractor's copy-paste. Writing down why a competitor blocks is inventing a fact about a company you have not spoken to, and it belongs nowhere in an evidence file.

Traffic consequences are equally off limits. Cloudflare's crawl-to-click analysis measured crawl-to-refer ratios varying enormously by platform in July 2025 (Cloudflare, 2025), which means the referral value of being crawlable is not a constant you can apply to someone else's site. Nobody controls answer-engine output, so no reading of anyone's robots.txt supports a forecast about their rankings, citations, or traffic — or yours.

Finally, do not publish the table with domains attached. We keep competitor crawl-posture tables internal and describe markets in aggregate, because a page naming a site as "blocking AI" is a claim about that business built on a file that may have changed the same afternoon.

Is a market where incumbents block a better market?

Our reading is yes, and we state it as reasoning rather than as a measured result [our data]. If the established sites in a niche disallow the search-index crawlers, they are not in the retrieval pool for those engines, which changes which sources are available to cite for that market's questions. We use that as one input to niche selection alongside demand and the state of the existing content — never as a projection of what any of it will produce.

The honest caveat, stated because it undermines our own use of the method: retrieval pools are not limited to a niche's obvious incumbents. General publishers, forums, and reference sites answer commercial questions constantly, and none of them appears in a competitor list built from the commercial SERP. A market with a blocked front page can still have a crowded retrieval pool.

Run the audit anyway — it is one request per domain, it uses only files their owners published, and it occasionally explains why a market looks emptier in AI answers than it does in search results. Then treat the output as one page of an evidence file, which is exactly the role it plays in the operating system this library documents.