User-Agent String

A user-agent string is the identification a client sends in its HTTP User-Agent header — the sentence a browser, crawler or script uses to say what it is. Robots.txt does not match that whole sentence. It matches a product token: a short name, restricted by RFC 9309 to letters, underscores and hyphens, that should appear as a substring of the full string.

The property that makes the rest of this library make sense is that the string is a claim, not an identity. Nothing in HTTP verifies it, and every rule you write in robots.txt is addressed to whoever chooses to answer to that name.

What is inside a user-agent string?

A product token, plus whatever context the operator decided to include — usually a browser-compatibility preamble and a URL explaining the bot. RFC 9309 asks that the identification string "describe the purpose of the crawler," and the AI vendors follow that convention. These are the strings as documented, read 2026-08-21:

OpenAI notes that the version numbers in these strings may change, which is the practical argument for matching on the token rather than the whole string. OpenAI also documents adding a robots.txt marker to the string when it fetches robots.txt itself, so site owners can separate that request from content requests in logs that do not record paths.

How does robots.txt match a user-agent?

By token, case-insensitively, to exactly one group of rules. RFC 9309 requires crawlers to use case-insensitive matching to find the group whose product token matches, so user-agent: gptbot and User-agent: GPTBot select the same rules. If more than one group matches the same token, the RFC requires those groups' rules to be merged into one.

Three fallbacks complete the logic, and they trip people in both directions. If no group matches the token, the crawler must obey the group with the * value. If there is no matching token and no * group — or no groups at all — no rules apply. And path matching runs on the opposite convention from token matching: the RFC says paths should be matched case-sensitively, starting from the first octet, with the most specific match winning and allow beating an equivalent disallow.

The consequence for a file you actually maintain: a * group is not a catch-all safety net for a bot you also named elsewhere. Name a token in its own group and that group is the only one it reads — which is exactly how a well-meaning "block everything, allow the good ones" file ends up allowing more than its author intended. Per-bot rule-writing is worked through in the robots.txt decision framework.

Why is a user-agent string a claim rather than an identity?

Because it costs one header to forge, and the popular bot names are the most useful camouflage on the web. A scraper sending GPTBot inherits whatever generosity your infrastructure extends to the real one: robots.txt allowances, WAF exceptions, rate-limit headroom.

The vendors treat it the same way, which is the strongest evidence for the point. OpenAI, Anthropic and Perplexity each publish crawler IP data specifically so operators can separate their bots from impersonators, and Google's crawlers overview documents identification by three signals — the user-agent header, the source IP, and the reverse-DNS hostname of that IP. The mechanics of running that check, and what to do when it fails, are in verifying real vs fake AI bots. The class of client doing the fetching is defined in web crawler.

Should you serve different content based on the user-agent?

No. Varying the page an engine receives from the page a person receives is cloaking, and Bing's Webmaster Guidelines list it among the abuse patterns that can cost a site visibility. The fact that the deciding input is a forgeable header should end the discussion on engineering grounds alone.

Reading the header is fine, and useful. Counting by token, rate-limiting a specific bot, or logging which agent fetched which URL are all ordinary operations. What breaks is branching the content on it — and there is no version of that trick that survives a spot check by anyone who curls your URL twice with two different headers. The page you want quoted should be the same page in every response, which is most of what our generative engine optimization guide argues for.

What should you do with the user-agent field in your logs?

Treat it as a label to group by, then verify before you believe it. Count fetches per token to see which platforms are working your site, keep the source IP on every line, and check membership against the vendor's published ranges before any number leaves your machine.

Against our own interest: for most sites this is a spot check, not a project. If you are not about to publish crawler statistics, rewrite robots.txt, or configure CDN blocks, glancing at the top tokens and top source IPs is proportionate — and no tool you can buy makes a self-reported string more trustworthy than it is.