AI Robots.txt Templates: Three Copy-Paste Patterns

Most AI robots.txt examples circulating are copy-paste lists with misspelled tokens, rules that conflict with a User-agent: * group, and Googlebot blocked by accident. This page is the implementation layer: three clean templates — allow all, block training only, block all — with every token verbatim from vendor documentation, and the syntax rules that decide whether your file does what it says. Which posture to pick is a business decision, made bot by bot in the robots.txt decision framework; this page assumes you decided and gets the file right.

Which AI crawler tokens exist to write rules for?

Twelve documented tokens from five companies, plus two significant undocumented bots — and token spelling is load-bearing, because a group aimed at a misspelled name matches nothing and fails silently.

Tokens per each vendor's crawler documentation as of August 2026; Bytespider and CCBot are undocumented by their operators and covered in their own pages. What each bot's job means for the blocking decision is the subject of GPTBot vs OAI-SearchBot, Anthropic's three crawlers, and the Perplexity agents.

What does the allow-everything template look like?

It contains no AI groups at all — under RFC 9309, a token with no matching group and no matching * restriction may crawl, so consent is expressed by silence:

# Template 1 — allow all crawlers (our posture, all 3 builds)
User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

This is what we run across our 3 production builds [our data], and the honest caveat is that it is a posture for sites that earn by being retrieved and cited. There is nothing clever to add: an empty Disallow: in the * group states the policy explicitly, and the Sitemap line is there because the file is fetched constantly anyway.

What does the block-training-only template look like?

One group per training-associated token, with the search and user-fetch tokens deliberately absent:

# Template 2 — block training, keep answer-engine visibility
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Corpus bots: open-archive and undocumented collectors
User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

The template's logic comes straight from the vendor docs: OpenAI documents GPTBot as its training crawler and OAI-SearchBot — absent here — as the index behind ChatGPT search, stating that opted-out sites "will not be shown in ChatGPT search answers." Google-Extended is a control token with no user agent of its own; disallowing it governs Gemini training and, per Google, does not affect Search inclusion or ranking. The corpus-bot groups are optional and stricter: CCBot feeds an open archive with diffuse downstream training uses, and Bytespider publishes no purpose statement at all.

What does the block-all template look like?

Every documented AI token gets a group — and Googlebot and bingbot deliberately do not, because both feed classic search first:

# Template 3 — block all documented AI crawling (needs a CDN backstop)
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

Know what this file can and cannot do before shipping it. The documented training and search crawlers state they honor robots.txt, so those groups are effective on the vendors' own terms. The ChatGPT-User and Perplexity-User groups are signals: OpenAI says robots.txt rules "may not apply" to user-initiated fetches, and Perplexity states its user agent "generally ignores" them — so a genuinely complete block requires CDN or WAF rules keyed to the IP ranges OpenAI, Anthropic, and Perplexity publish in the same documentation that defines the tokens. This template also costs you answer-engine visibility by design; if that surprises you, the decision page — not this one — is where to start.

What syntax mistakes break AI robots.txt files?

The most-specific-match rule causes nearly all of them. Under RFC 9309, a crawler that finds a group matching its own token uses only that group — so once GPTBot has a group, your User-agent: * rules say nothing to GPTBot, in either direction. The two recurring bugs: a strict * group that everyone assumes covers AI bots which each have their own permissive group, and the reverse, where adding one AI-specific group silently exempts that bot from sitewide path restrictions it used to inherit.

Three smaller rules round out the checklist. Groups are matched per token, and rules never combine across groups, so write each group as if it were the only one the bot will read — because it is. Crawlers must be able to parse at least 500 kibibytes of robots.txt per RFC 9309, so a long per-bot file costs nothing. And Crawl-delay is not in the RFC — among the AI vendors, only Anthropic documents honoring it, so use it inside Anthropic groups and expect nothing from it elsewhere.

Last, verify the served file, not the repo copy. We found a CDN-managed robots.txt silently overriding a build's allow-everything file on a production zone [our data] — one curl https://yourdomain.com/robots.txt compared byte-for-byte against your source is the cheapest audit in this whole stack. What you are ultimately tuning — pages that earn the retrieval these rules permit — is covered in our generative engine optimization guide.