What Blocking AI Crawlers Costs — and Saves

The block-or-allow debate usually skips its own arithmetic. One side prices the crawling ("they take 38,000 pages per visit!"), the other prices the citations ("you'll vanish from AI answers!") — and both sides quote numbers that only cost or pay anything under specific architectures. This page does the pricing: what allowing costs, what blocking saves, what blocking forgoes, and the design rule that moves most sites' answer to "the money doesn't care" [our data]. The decision itself — which bots, which business models — lives in the publisher block-or-allow framework; this page is the arithmetic under it.

What does allowing AI crawlers actually cost?

On static architecture, close to nothing. Our 3 production builds serve statically generated pages from a CDN; the hosting line is $0 a month on two builds and $20 on the third, and it would be the same line at 10x the crawler volume [our data]. A crawler fetch of a static page is the cheapest thing a website can do.

The volume is real even when the cost isn't. AI crawler fetches vastly outnumber the AI-assistant referral sessions our analytics records — the same lopsided shape Cloudflare measures network-wide [our data]. What that imbalance means, and how to compute your own, is the crawl-to-refer ratio; the July 2025 snapshot (Cloudflare, published August 2025) frames the spread:

The table prices nothing by itself. It tells you how much serving you are donating — and whether that donation costs money depends entirely on what a request triggers on your stack, which is the next two sections.

What does blocking actually save?

Whatever your per-request costs are, multiplied by crawl volume — which for most content sites is a small number multiplied by a large one. Bandwidth and compute on CDN-served static pages round to zero, so blocking saves approximately that: zero [our data]. Blocking-for-bandwidth mostly saves a statistic, not a bill.

The savings become real in two situations. First, metered backends: pages whose requests trigger server rendering, database work, or per-call vendor APIs turn crawl volume into spend, and a 4-orders-of-magnitude spread in crawler enthusiasm (see the table above) means the bill depends on who finds you. Second, genuinely constrained origins — self-hosted infrastructure where aggressive fetching degrades service for humans. In both cases enforcement is its own line item: robots.txt is voluntary under the Robots Exclusion Protocol, and the harder backstop is CDN-layer controls of the kind Cloudflare documents for AI training crawlers (Cloudflare, content controls). Which bots to point those controls at is cataloged in the complete AI crawler list.

What does blocking forgo?

Citation exposure — priced honestly, in two currencies. In clicks, the forgone asset is small: Pew measured users clicking a source cited in an AI summary in roughly 1% of summary visits, against an 8% versus 15% overall link-click rate with and without a summary present (Pew, March 2025 data). Blocking every AI crawler on a typical content site turns off a channel that was never large.

In presence, the forgone asset is harder to dismiss. Where citations do land, they correlate with better outcomes on the same SERP — pages cited in an AI Overview saw about 35% higher CTR in Seer's 2025 data — and an uncrawlable site cannot be quoted, recommended, or corrected in the surfaces where a growing share of buyers ask their questions. That value never registers as a session, which is exactly why it gets priced at zero by people arguing from analytics alone. We price it above zero: our own fleet allows the crawlers because we sell being found, and a citation is a free impression in the answer layer [our data].

How do you keep crawler traffic from costing anything?

Design the meter out of the request path — then the whole cost side of this page collapses to zero and the decision becomes pure strategy. The rule from our car-leasing build, which carries a metered market-data API under a heavily crawled deals surface: cache markets whole, refresh them on a schedule priced against the vendor's per-call rates, and serve every visitor — human or bot — from the cache. API usage then scales with markets times refresh cadence, never with visitors, so a crawler IP storm cannot multiply the bill [our data].

The general form works on any stack: static-render everything renderable, cache whole datasets instead of querying per request, and let scheduled jobs — not inbound traffic — be the only thing that touches metered services. A site built this way can afford to make its blocking decision on the framework's terms — bot class by bot class, business model first — without a budget thumb on the scale.

So is blocking worth it, in dollars?

For a static content site: no meaningful savings, a small-in-clicks and real-in-presence forgone asset, and enforcement effort on top — which is why our fleet allows and why sites whose content is the product rationally block anyway, for rights reasons the arithmetic doesn't capture. For metered architectures: the savings are real until the design rule above removes them, and removing them is usually cheaper than the blocking debate. Run your own numbers before borrowing anyone's — including ours: count what a crawler request actually triggers on your stack, price it, and only then argue strategy, ideally with the operator's guide open to what being quotable is worth.