What Is Google-Extended — and Should You Block It?

Google-Extended is the most misread line in AI-era robots.txt files. Operators paste it into block lists expecting to disappear from AI Overviews, or leave it alone fearing a ranking penalty — and per Google's own documentation, both expectations are wrong. This page states what the token actually governs, quotes the documentation for each boundary, and gives the decision the way we make it on our own builds.

What is Google-Extended?

Google-Extended is a robots.txt control token, not a crawler. Google's crawler documentation describes it as "a standalone product token" that publishers use to manage whether content Google crawls from their sites may be used for "training future generations of Gemini models" and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.

The mechanical detail that confuses everyone: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings." No bot named Google-Extended will ever fetch a page from you. Googlebot and its siblings do the fetching exactly as before; the token only changes what Google permits itself to do with what it fetched.

That makes Google-Extended different in kind from GPTBot or ClaudeBot, which are real crawlers you can count in logs. It is one line of policy, addressed to one company, enforced entirely inside that company's pipeline — the full roster of actual bots is in the complete AI crawler list.

What does disallowing Google-Extended actually stop?

Two documented uses, both outside Google Search: training future Gemini models, and grounding answers in Gemini Apps and Grounding with Google Search on Vertex AI. Everything else operators attribute to the token is projection. The boundary map, per Google's crawler and AI-features documentation as of August 2026:

Google's AI-features page draws the same line from the other side: it points publishers to Google-Extended "to limit AI training and grounding in some of Google's other systems" — other systems, as distinct from the Search features the page is about.

Does blocking Google-Extended remove you from AI Overviews?

No — and this is the mistake that costs sites nothing but teaches them the wrong lesson. Eligibility for AI Overviews and AI Mode has exactly 1 documented rule: a page must be indexed and eligible to be shown in Google Search with a snippet. The levers that limit those features are the snippet controls — nosnippet, data-nosnippet, max-snippet, and noindex — each documented in Google's robots meta tag reference, and each also constraining classic results.

So a site that disallows Google-Extended and then sees itself cited in an AI Overview has not caught Google ignoring robots.txt. It has demonstrated the documented boundary: AI Overviews are a Search feature, fed by Search's index, governed by Search's controls. The two generative surfaces and their shared eligibility rule are unpacked in what Google AI Mode is.

How do you block Google-Extended in robots.txt?

One group, two lines, using the token verbatim:

User-agent: Google-Extended
Disallow: /

Partial rules work the same way as for any token — disallow a directory instead of / to scope the opt-out. Two operational notes. First, there is nothing to verify in your logs afterward: no user agent changes behavior, because none exists. The file itself is your only record. Second, the token is honored at the policy layer, like every robots.txt rule — it is a published instruction to one company about permitted use, not an access control.

Should you block Google-Extended?

Decide it as a training-posture question, because per Google's documentation that is the only thing at stake. The token is unusual among AI opt-outs: blocking GPTBot or ClaudeBot at least changes observable crawl traffic, while blocking Google-Extended has no stated cost to Search inclusion, ranking, or AI-feature eligibility — and no observable effect at all outside Google's own systems. That makes it the cheapest "no" a publisher can say to AI training.

On our 3 production builds we leave it open [our data]: our sites earn from being visible and accurately represented wherever answers get assembled, so Gemini training and grounding access is consistent with how we make money. Against our own interest here: if you license content, sell subscriptions, or otherwise treat your archive as the asset, disallowing Google-Extended is the rare opt-out with no documented visibility bill attached, and we would take it in your position. The same job-by-job logic for every other token is in the robots.txt decision framework.

One honest caveat closes the decision. The token governs exactly what Google's documentation says it governs, and that scope is Google's to define — which is why we re-verify the crawler docs quarterly rather than trusting any summary, including this one, indefinitely. What being retrievable and citable is actually worth — the reason most sites leave every door open — is the subject of our generative engine optimization guide.