AI Training Data

AI training data is the body of content a model learns from before it answers anything: the corpus used to fit its parameters, frozen at a cutoff when training ends. It is not the same thing as the index an engine searches while composing an answer — and treating the two as one is the most expensive vocabulary error in this field.

This library uses the distinction on page after page without stopping to define it. Here it is once: two corpora, collected by different crawlers, controlled by different robots.txt tokens, with different consequences for what an assistant says about you.

How is training data different from a search index?

Training data is absorbed in advance and frozen; a retrieval index is queried live, per question. That difference decides whether anything you publish this quarter can reach a given answer at all.

Vendor crawler documentation, read 2026-08-21.

Perplexity marks the boundary from the other side, stating that PerplexityBot "is not used to crawl content for AI foundation models" — a search crawler that explicitly disclaims the training job. Google's case is the odd one out: crawling happens under existing Google agents, and Google-Extended is a control token rather than a bot, which is why it never appears in logs. That token's exact scope is covered in what Google-Extended is.

Does one robots.txt rule control both corpora?

No — and OpenAI says so in the plainest sentence any vendor has published on the subject: "Each setting is independent of the others," with the worked example of allowing OAI-SearchBot to appear in search results while disallowing GPTBot so that crawled content is not used for training. Anthropic splits the same way, documenting ClaudeBot for training and Claude-SearchBot for search quality as separate tokens with separate consequences.

The failure mode this creates is blanket blocking. "Block AI bots" as a category rule takes out the search crawlers along with the training crawlers, which is how a site quietly removes itself from ChatGPT search answers while trying to protect its content from model training. Which token to allow, per business model, is decided in should publishers block AI crawlers? and, for the OpenAI family specifically, in GPTBot vs OAI-SearchBot.

If you block a training crawler now, does your content leave the model?

No vendor documents any mechanism for that. Anthropic's wording is forward-looking: restricting ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets." OpenAI's is the same shape — disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models."

Read those carefully and the limit is clear. A block is a preference about future collection, not a deletion request, and models already trained on a page stay trained on it. Anthropic adds one operational note worth repeating: blocking by IP instead of robots.txt "may not work correctly," because it also blocks the crawler from reading the robots.txt file that carries your preference.

Why does the distinction change what an assistant says about your brand?

Because only one of the two corpora is reachable by publishing. When an engine retrieves, your current page can be found, quoted and linked. When a model answers from training alone, it is describing the version of your brand that existed in its training window — and nothing you publish this month is in the loop. That split, and what each side means for corrections, is defined in grounding.

The diagnostic that follows is cheap and worth running before any expensive fix: when an assistant states something wrong about you, first establish whether the wrong answer came with citations. Cited answers are retrieval problems, addressable by publishing. Uncited ones may be training-era artifacts, on a timeline nobody outside the model provider controls.

How many sites actually opt out of AI training?

Very few, which makes the debate louder than the practice. Cloudflare's July 2025 analysis (Cloudflare, July 2025) found only about 37% of the top 10,000 domains had a robots.txt file at all; among the files it did find, GPTBot was disallowed in 7.8% and Google-Extended in 5.6%, with anthropic-ai, PerplexityBot, ClaudeBot and Bytespider each under 5%.

We are part of the majority here, and we say why rather than presenting it as a recommendation. Across our 3 production builds we allow the documented AI crawlers, training crawlers included [our data], because our sites earn from being visible and accurately represented wherever answers get assembled. Against our own interest: that calculus flips for anyone whose archive is the product. A publisher licensing content, or selling subscriptions to it, has a real asset to protect and a weak return from training access — and per the vendors' own documentation, the training opt-out and the search opt-out are separate lines you can write independently. What eligibility on the retrieval side is actually worth is the subject of our generative engine optimization guide.