Server Log Analysis: Which AI Bots Crawl Your Site

Server logs are the one place AI crawler activity is a fact instead of a guess — and almost nobody who writes about AI bots appears to read any. This is the procedure we run on our own three production builds: which tokens to search for, how to avoid counting impostors, and what the numbers do and do not mean once you have them.

Why can't analytics show AI bot traffic?

Because the documented AI crawlers fetch raw HTML and do not execute JavaScript — and analytics tools only see visitors that run their script. A GPTBot fetch of your page produces a log line on your server and nothing at all in GA4. The two systems measure different populations: analytics sees humans (including humans referred by AI assistants), logs see every request, bots included.

So the toolchain for this question is unglamorous: your web server's access logs, or your CDN's request logs, plus grep. If you retain logs for 30 or more days you can also read trends, which is where the actual findings live.

Which user agents should you search for?

Eight documented tokens from the three AI platforms that publish crawler documentation. Copy them exactly — a robots or grep rule aimed at a misspelled token matches nothing, which is why we never paraphrase a token name:

Googlebot and bingbot belong in the same review (both feed AI answer surfaces), and OpenAI has added tokens before — vendor docs are the live authority. The full 12-token roster across 5 companies, including Google-Extended's special status as a control token, is maintained in the complete AI crawler list.

How do you pull and count the fetches?

Two commands cover most of the job on a standard combined-format log. Count requests per token:

for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User \
          Claude-SearchBot PerplexityBot Perplexity-User; do
  printf '%-18s %s\n' "$ua" "$(grep -c "$ua" access.log)"
done

Then, for any token with meaningful volume, list what it actually fetched:

grep 'ClaudeBot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Practical notes from running this on our own fleet: bucket counts by week (log filenames or timestamps) so you get a trend, not a trivia number; run the path report per token, because what a training crawler fetches broadly and what a user-triggered fetcher requests specifically are different findings; and if you are behind a CDN, prefer its logs — origin logs undercount whenever the CDN serves cached copies without touching your server.

How do you verify the bots are real?

Check source IPs against the vendors' published lists, because the user-agent string proves nothing — any scraper can send GPTBot as its user agent, and spoofed AI user agents are common enough to contaminate an unverified analysis. The documented verification paths: OpenAI publishes IP ranges for its bots in the same documentation that defines them; Anthropic and Perplexity publish JSON files of official crawler IPs linked from their crawler docs; Google documents checking the user agent, the source IP, and the reverse-DNS hostname for its crawlers.

The lazy-but-sound version of this step: spot-check the top source IPs for each token against the vendor's list before believing any count. The rigorous version — filtering every hit — is what turns a log skim into data you would publish.

What do the counts actually tell you?

Less than the raw totals suggest, and the published baseline explains why. Cloudflare's crawl-to-click analysis measured how many pages each platform crawls per referral click it sends back: roughly 38,066:1 for Anthropic, 1,091:1 for OpenAI, and 195:1 for Perplexity in July 2025, against about 5.4:1 for Google. Heavy fetch volume from a training crawler is a corpus-building signal, not evidence any engine is about to cite you — which is why step 5 of the procedure classifies every token by job before reading meaning into its count. The metric itself is defined at crawl-to-refer ratio.

What running this on our own three production builds shows: the documented crawlers — ClaudeBot, GPTBot, PerplexityBot among them — fetch our HTML pages routinely [our data]. The same discipline produces the running count in our in-progress 90-day llms.txt watch: 0 fetches of /llms.txt by any documented AI engine crawler across all 3 builds so far, while those same crawlers fetched HTML throughout — an interim tally, since no build is yet 90 days old. We have not yet published dated per-bot fetch tables from the fleet; when we do, they will be log extracts with periods attached, not estimates [our data].

What should you change based on the analysis?

Robots policy, per token, with your own numbers finally in hand — that is the payoff of the whole exercise. Set each token's verified fetch count against what its platform sends back (your AI-referral sessions, from analytics), and you have a per-platform crawl-to-refer ratio for your site instead of an industry average. A pageview-monetized publisher staring at a five-figure ratio has a rational case to block training crawlers; a lead-generation or authority site — our situation, and why we currently allow the documented crawlers — mostly benefits from being retrievable [our data].

The business-model decision table lives in should publishers block or allow AI, and the page-structure work that makes the allowed fetches worth something is our generative engine optimization guide. Whatever you decide, decide it from your logs — the one dataset on this topic nobody can launder on your behalf.