CCBot: What Blocking Common Crawl Actually Does

CCBot is the most misunderstood line in the AI section of robots.txt, because it is the only major AI-associated crawler that does not belong to an AI company. Block GPTBot and you have made a decision about OpenAI. Block CCBot and you have made a decision about an open corpus and every pipeline that draws on it — most of which you cannot name, and none of which will ever tell you what changed. This page lays out what is measured, what is mechanical, and what is honestly unknowable about that decision.

What is CCBot?

CCBot is the web crawler of Common Crawl, a nonprofit that publishes an open corpus of crawled web data, free for anyone to download and use. That structure is the entire story: CCBot does not feed one company's model or one engine's index — it feeds a public archive, and the archive's users range across AI training, academic research, and whatever else a downloadable copy of the web enables.

One sourcing note, stated plainly because this niche rarely states them: no Common Crawl documentation is on our closed source list. So this page confines its claims to what Cloudflare measures, what RFC 9309 defines, and what our own infrastructure record shows [our data] — and marks corpus specifics like snapshot size, crawl cadence, and the exact roster of downstream users as unverified rather than repeating numbers we cannot check.

How much does CCBot actually crawl?

Very little, by the one published measurement on our source list — and the volume is stable, not growing. Cloudflare's crawl-to-click study puts CCBot's share of AI-bot traffic on its network at 0.1% in July 2024 and the same 0.1% in July 2025, a footprint two orders of magnitude below the platform training crawlers.

Period and figures per Cloudflare's crawl-to-click analysis; platform crawlers omitted here — the full roster is in the complete AI crawler list.

The practical upshot: server load is a weak argument anywhere in this decision. Whatever you decide about CCBot, you are deciding about corpus inclusion, not bandwidth.

What happens if you block CCBot?

Your pages stop entering future Common Crawl snapshots — and every consequence after that is diffuse, downstream, and invisible from where you stand. A model trainer who filters the corpus next year will not see your domain. A researcher building an evaluation set will not sample your pages. None of them will notify you, none of their choices appear in any analytics property you own, and no crawler dashboard has a column for "answers your absence shaped."

This is the structural difference between CCBot and every platform crawler. Blocking GPTBot has a bounded, documented effect you can reason about — one company's training pipeline, per OpenAI's own docs. Blocking CCBot has an unbounded, undocumented effect: you are editing yourself out of a shared input that an unknown number of current and future systems draw from, the same corpus-shaped substrate that retrieval-based answer engines sit on top of. We can state the direction of the effect honestly; nobody can state its size, and you should distrust anyone who quotes one.

Two boundaries keep the decision honest. Blocking is not retroactive — published snapshots and already-trained models keep whatever they collected. And it is not a substitute for platform decisions: GPTBot, ClaudeBot, and peers crawl independently of Common Crawl and each need their own line.

Does CCBot respect robots.txt?

Common Crawl is widely described as honoring robots.txt, but no source on our closed list documents its compliance policy — so we treat compliance as reported rather than verified, and we say so instead of rounding it up to a fact. The rule itself is ordinary RFC 9309 syntax:

User-agent: CCBot
Disallow: /

Under RFC 9309, robots.txt rules "are not a form of access authorization" — for any crawler, compliance is the operator's choice, and enforcement beyond trust is a CDN or WAF job. In our fleet's log reviews the platform crawlers — ClaudeBot, GPTBot, PerplexityBot — are the regular AI visitors we track and verify; we have not published dated CCBot fetch counts, and when we publish per-bot numbers they will be log extracts with periods attached, not estimates [our data].

Are you blocking CCBot without knowing it?

Possibly — CCBot is 1 of the 8 tokens Cloudflare's managed robots.txt disallows, and we have seen that default live in production. During a pre-cutover audit on our auto-finance authority rebuild, Cloudflare was auto-serving a 2,895-line robots.txt with Disallow: / for 8 AI crawlers, CCBot among them, while the robots.txt in the build's git repo said allow [our data]. Nobody had decided to block Common Crawl; a dashboard default had.

The check takes one minute: fetch https://yourdomain.com/robots.txt and read what is actually served, not what your repo or CMS contains. If a CDN sits in front of your site, the served file is the only one that counts.

Should you block CCBot?

It is the rare AI-crawler call where both answers are defensible, because the trade is licensing philosophy rather than measurable traffic. A publisher whose content is a licensed asset has a coherent case for blocking: an open, freely downloadable corpus is the least controlled form of distribution that exists, and 0.1% of bot traffic buys none of it back. A business that earns by being widely known — our situation — has the opposite case: the corpus is one more path into the systems that answer questions about your topic.

We allow CCBot across all 3 production builds [our data], consistent with the allow-everything posture argued in should publishers block or allow AI crawlers. Against our own interest: if your revenue depends on controlling copies of your content, our posture is wrong for you, and this is the block to make before any platform-specific one. Either way, decide it on purpose — and if you leave the door open, the pages behind it should be worth quoting, which is the entire subject of our generative engine optimization guide.