Googlebot

Googlebot is Google's web crawler for Search: the client that discovers URLs, fetches them, and feeds the index Google serves results from. Google classifies it as a common crawler, the category its documentation says "always respect[s] robots.txt rules for automatic crawls."

It earns a place in an AI-answer glossary for one reason. Google's AI surfaces are documented as running on Search, so Googlebot's access is upstream of whether AI Overviews and AI Mode can use a page at all.

Why does Googlebot decide what AI Overviews and AI Mode can use?

Because Google documents its AI features as part of Search rather than as a separate system with its own crawler. Its AI-features page states it directly: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search."

The eligibility rule attached to that is a single sentence — a page must be indexed and eligible to be shown in Google Search with a snippet, with "no additional technical requirements." Google's generative-AI optimization guide describes the mechanism behind it as retrieval-augmented generation "relying on our core Search ranking systems to retrieve relevant, up-to-date web pages from our Search index."

Two boundaries follow, and both are commonly misread. Blocking Googlebot removes a site from Search and therefore from the surfaces Search feeds. Blocking Google-Extended does something else entirely — it governs training and grounding in Google's other systems, which is unpacked in what Google-Extended is. What actually gets a page selected once it is eligible is a different question again, covered in how to show up in AI Overviews.

Does Googlebot process JavaScript?

Google says it can, in the one closed-list document that addresses it. Its generative-AI optimization guide states that "Google is able to process content within JavaScript as long as it isn't blocked," while adding that working on SEO with JavaScript frameworks is generally more complex and pointing to its JavaScript SEO guidance.

That statement is Google's, about Google, and it does not travel. No AI platform's crawler documentation makes an equivalent claim about its own bots, which is why our own engineering rule is unchanged: serve the answer in the HTML response, and keep JavaScript for interaction. The evidence behind that rule — and how thin it is on the AI-crawler side — is in do AI crawlers render JavaScript?.

What technical limits does Google document for its crawlers?

More than most operators expect, and the numbers are specific. These are the properties Google publishes for its crawling infrastructure, read 2026-08-21:

Two of these are worth acting on. The 15MB cut is a real ceiling on bloated HTML documents, and the caching headers are free crawl efficiency that most CMS setups already emit — Google recommends setting both ETag and Last-Modified regardless of its own preference, because other systems use them.

Should you ever block Googlebot?

Almost certainly not, and the decision is different in kind from the AI-crawler decisions this library spends most of its time on. A publisher weighing GPTBot or ClaudeBot is trading crawl access against training use with little traffic returned. Blocking Googlebot removes a site from Google Search — and, per Google's own documentation, from the AI features that run on it.

We allow it across all 3 of our production builds [our data], which is an unremarkable position rather than a finding. The honest caveat sits on the other side of the trade: access buys eligibility, not outcomes. Google states that meeting every requirement, best practice and policy still "doesn't mean that Google will crawl, index, or serve its content," and nobody — us included — can promise more than that. What a crawler is and what its fetch actually proves is defined in web crawler; what you do with eligibility once you have it is our generative engine optimization guide.