How to Verify Real vs. Fake AI Bots in Your Logs

Every AI-crawler decision downstream — counts, ratios, robots.txt rules, CDN blocks — inherits its quality from one unglamorous step: checking that the bot in your logs is who it claims to be. The user-agent string is a text field anyone can populate, impersonation is routine, and all the log analysis in the world is fiction if a scraper wearing "GPTBot" is in the sample. This is the verification procedure we run on our own fleet, using only the checks the vendors themselves document — including the lesson we earned by doing the active-testing part wrong once [our data].

Why is the user-agent string not enough?

Because it is self-reported: any HTTP client can send GPTBot as its user agent, and the popular bot names are exactly the camouflage a scraper wants. Impersonating a known AI crawler inherits whatever generosity your infrastructure extends to the real one — robots.txt allowances, WAF exceptions, rate-limit headroom — at the cost of editing one header.

The vendors say the same thing, which is why the verification data exists. The three AI platforms with documented crawler families all publish IP information precisely so operators can separate their bots from impersonators; a count that skips the check measures "whoever felt like claiming the name," not the platform. Where each token fits in the documented roster is in the complete AI crawler list.

What verification data does each vendor publish?

Three AI vendors publish IP data, and Google documents the strictest recipe — per the official docs, current as of August 2026:

Fetch the lists fresh at verification time rather than vendoring them into a script last touched in a previous quarter: ranges change, and a stale list quietly turns real crawlers into "impostors" in your reports. The bottom row is its own finding — an unverifiable bot can never be distinguished from its impersonators, one more consequence of Bytespider's missing documentation.

How do you run the check, mechanically?

Filter, fetch, and test containment — the six steps in the box above, with two technicalities that decide whether the result is trustworthy. First, test IP membership with real CIDR arithmetic (grepcidr, ipcalc, or Python's ipaddress module), never with substring matching: an octet-level grep happily places 203.0.113.5 inside an unrelated range and misclassifies in both directions. Second, for Googlebot use the round trip Google documents — reverse-resolve the source IP to a hostname, then forward-resolve that hostname and require the original IP back — because reverse DNS alone is spoofable by whoever controls the reverse zone.

The volume-counting layer that this feeds — which tokens to grep, how to bucket by week, what the totals mean — is its own procedure in server log analysis for AI bots. Run verification before believing any number that procedure produces; filtering to verified IPs is the difference between measuring AI crawler behavior and measuring whoever impersonated it that week.

What do you do when verification fails?

Reclassify the traffic, and protect your numbers from it — a failed check is a positive finding, not noise to shrug at. The hit is ordinary unidentified scraping: exclude it from every AI-crawler count, keep it out of your crawl-to-refer ratio arithmetic, and hand it to your normal bot policy, whether that is rate limiting, blocking, or indifference.

What you must not do is attribute impostor behavior to the platform it named. "ClaudeBot ignored my robots.txt" is a serious claim about a vendor's documented compliance — and the commonest version of it we encounter dissolves the moment source IPs are checked. Verify first; accuse second, if at all.

What is the trap in active testing?

Getting your own IP penalty-boxed — at which point every subsequent test from that machine returns garbage while looking like data. We learned this on our auto-finance migration build: during a pre-cutover audit, a couple dozen unpaced verification curls got the operator machine 403'd for everything, including requests with a normal Chrome user agent, poisoning each later test from that vantage until the work moved to a clean IP [our data].

Two rules came out of that afternoon, and they now govern all our active testing [our data]. Pace probes about 60 seconds apart and make one pass — enough to answer the question, never enough to look like an attack; separating a real per-UA block from rate-limit noise on that same audit took interleaved fetches spaced about 70 seconds apart. And treat your own machine's results as provisional however clean the pacing: your requests do not originate from the crawlers' published ranges, so the final proof that real bots get through is an origin log line — documented UA, verified source IP, status 200.

Against interest, honestly: most sites never need the rigorous version of any of this. If you are not about to publish crawler statistics, change robots.txt, or configure CDN blocks on the strength of your logs, a spot check of the top source IPs per token is proportionate — the full procedure exists because we publish what our logs show, and unverified logs are not worth publishing. The reason any of it matters — being fetched by real engines and quoted in their answers — is the subject of our generative engine optimization guide.