Web Crawler

A web crawler is an automated client that fetches URLs, reads what comes back, and follows links to discover more URLs. RFC 9309 defines crawlers as automated clients and specifies the robots.txt rules they are asked to honor; Google's crawler documentation calls crawler "a generic term for any program that is used to automatically discover and scan websites."

This is the base term the rest of this library leans on. Every AI-bot page, every robots.txt decision, and every log analysis assumes it — so it gets defined once, here, rather than half a paragraph in twenty places.

What does a web crawler actually do?

It requests a URL, stores or processes the response, extracts the links it finds, and queues those for later requests. RFC 9309 puts the loop plainly: search engines "have crawlers to recursively traverse links for indexing." Everything else — scheduling, deduplication, politeness delays, retry logic — is engineering around that loop.

Two properties matter for a publisher. A crawler is an ordinary HTTP client, so it sees exactly what your server returns to that request, not what a browser assembles afterwards. And it identifies itself in a header it fills in itself, which is why identity is a separate problem from access; the string and its matching rules are covered in user-agent string.

How is a crawler different from a fetcher?

A crawler runs on its own schedule and traverses links; a fetcher makes one request because a person asked for it. Google draws that line in its own documentation — fetchers "act as a program like wget that typically make a single request on behalf of a user" — and then sorts all of its clients into three categories:

Google crawler documentation, read 2026-08-21.

That three-way split is not a Google quirk. The AI platforms use the same shape — an automatic crawler, and a fetcher that runs when a user asks an assistant to open a page — and it is why "should I block AI bots?" has no single answer. "Scraper," by contrast, appears in no specification on our source list; it describes intent, and nothing you can match a rule against.

What rules is a crawler expected to follow?

Whatever the site publishes in /robots.txt, on the honor system. RFC 9309 specifies the file's syntax, where it lives, and how a crawler selects the group that applies to it — and states directly that these rules "are not a form of access authorization." A crawler that ignores them is not breaking a lock; it is ignoring a sign.

The RFC also sets the operational edges. Crawlers should follow at least five consecutive redirects to reach the file, should not use a cached copy for more than 24 hours, and should impose a parsing limit — one that must be at least 500 kibibytes. When the file returns a 4xx status the crawler may access any resource on the server; when it returns a 5xx the crawler "MUST assume complete disallow" — a detail worth knowing before your CDN starts erroring under load. Which rules to actually write for which bot is the subject of the robots.txt decision framework.

Is being crawled the same as being indexed or quoted?

No — they are three separate outcomes, and conflating them is the most expensive mistake in this whole vocabulary. A crawl is a fetch. Indexing is a system deciding to keep and organize what it fetched. Being quoted is a third system selecting your passage while composing an answer. Google states the boundary against its own interest: meeting every requirement, best practice and policy still "doesn't mean that Google will crawl, index, or serve its content."

The practical version: a spike of AI-crawler fetches in your logs is evidence of access, not of visibility. We treat crawl counts as an eligibility signal and nothing more, which is also why we never read heavy bot traffic as a sign that a page is being cited. What actually decides whether a fetched page competes at answer time runs through our generative engine optimization guide.

Which crawlers show up on a site like yours?

The documented AI crawlers, the search engines' crawlers, and a long tail of unidentified clients. Across our 3 production builds, ClaudeBot, GPTBot and PerplexityBot appear in our server logs as regular visitors [our data] — an observation from routine log review, not a dated fetch table, and we have not published per-bot counts from the fleet.

Two cautions before anyone acts on a log file. Analytics cannot see any of this, because analytics runs on JavaScript that crawlers do not execute; server or CDN logs are the only source. And the user-agent field is self-reported and costs one header to forge, so any line reading "GPTBot" may be somebody else wearing the name — we have not measured how often that happens on our own sites, and neither has anyone whose numbers we would trust. The token inventory is in the complete AI crawler list, and the counting method — grep, bucket, verify — is in server log analysis for AI bots.

Against our own interest: most site owners never need this taxonomy. It earns its keep only when you are about to write a robots.txt rule, read a log, or price crawler traffic — and if you are doing none of those this quarter, the correct amount of attention to pay your crawlers is none.