User-Triggered Fetcher

A user-triggered fetcher is an automated client that requests a page because an end user asked for it, rather than on a crawling schedule. Google names the category in its own crawler documentation: "User-triggered fetchers are part of tools and product functions where the end user triggers a fetch."

It is the class this library keeps invoking without defining — the one that separates a bot collecting a corpus from a bot answering somebody's question right now.

How is a user-triggered fetcher different from a crawler?

By what starts it and how far it goes. A crawler runs on its own schedule and traverses links to discover more URLs; a fetcher makes one request because a person asked. Google's documentation puts the difference in the behavior — a fetcher acts as "a program like wget that typically make[s] a single request on behalf of a user" — and sorts all of Google's own clients into three categories, of which user-triggered fetchers are one. Google Site Verifier is the example its documentation names.

The base class and Google's full three-way split are defined in web crawler. What matters here is the consequence: everything you know about crawl budgets, politeness delays and scheduled revisits describes the other categories. A user-triggered fetch has none of those properties. It happens once, when someone types something.

Which AI agents are user-triggered fetchers?

Three, one per vendor, each documented by the company that runs it. Vendor documentation read 2026-08-21:

Sources: OpenAI, Anthropic and Perplexity crawler documentation. The full inventory of documented AI tokens, including the training and search-index crawlers this page deliberately leaves out, is the complete AI crawler list.

Do user-triggered fetchers follow robots.txt?

Not consistently, and this is the only bot category where the answer changes by vendor. Two of the three vendors above document their fetcher as standing outside normal robots.txt handling because a human initiated the request; the third documents no such exception.

Two practical consequences follow. A robots.txt rule aimed at these agents is not portable — the same three lines produce three different behaviors, per the vendors' own words, so a rule that works as intended for one is a statement of preference for another. And any measurement built on the assumption that robots.txt gates all bot traffic is wrong at the edges by an amount you cannot see from inside your own logs.

Note also what Google's crawlers overview does not say. It defines the category and names an example; it does not state how user-triggered fetchers treat robots.txt. That claim is sometimes attributed to Google's documentation, and it is not on the page this library cites.

What does a user-triggered fetch actually tell you?

That one person asked one question, and an assistant went and looked at your page while answering. That is a genuinely different signal from a training crawl, and it is the reason this category is worth defining separately: it is the only crawler class that correlates with a live human being interested in your subject at that moment.

It is also weaker evidence than it feels like. The fetch does not reveal whether the answer quoted you, linked you, or discarded the page as irrelevant — nothing in a server log distinguishes those outcomes. Volume across the whole bot population is a different metric again, measured as the crawl-to-refer ratio. Our own published log observations name the training and search crawlers we see as regular visitors; we have not published per-token counts for the user-triggered agents on any build, and will not until they are dated log extracts rather than impressions [our data].

Should you block a user-triggered fetcher?

That is a business decision, not a hygiene default, and it is a different decision from blocking a training crawler. Blocking a training crawler withholds your content from a future model corpus. Blocking a user-triggered fetcher withholds a page from someone who is asking about your topic right now — which is the traffic most sites are trying to earn.

The per-token decision, including how the training and search-index jobs differ, belongs to GPTBot vs OAI-SearchBot; this page only defines the class. Against our own interest as people who build sites to be retrieved: if you are not about to edit robots.txt, read a log, or price crawler access this quarter, the correct amount of attention to pay this distinction is none. Where it does fit into a visibility system is mapped in our generative engine optimization guide.