How to Rate Limit AI Crawlers Without Losing Citations

The AI-crawler conversation is usually binary: block them or let them in. There is a third posture — serve them, slower — and it is the one with almost no documentation behind it. This page collects what is actually published about crawl rate, names the failure mode that turns a throttle into a site-wide block, and argues that most sites should fix the cost instead of the traffic.

Can you rate limit AI crawlers instead of blocking them?

You can try, but only one AI platform documents a mechanism for it, so most of what you configure is behavior nobody has specified. Here is every crawl-rate statement available across the operators we track, read 2026-08-21:

One of the three AI-specific operators publishes a throttle. The search engines, which have been managing crawl budgets for decades, both publish one. That gap is the honest state of the art, and it is why this page cannot tell you what a 429 does to your citation odds — nobody has said.

What does the robots.txt standard actually say about crawl rate?

Nothing. RFC 9309 standardizes three record types — user-agent, allow and disallow — and explicitly leaves everything else to implementers: "Crawlers MAY interpret other records that are not part of the robots.txt protocol." Crawl-delay is one of those other records. It has no standardized meaning, no standardized units, and no obligation attached to it.

That reframes the directive correctly. Writing Crawl-delay: 5 is not configuring anything; it is leaving a note for software that may or may not read it. For ClaudeBot the note is documented as read. For every other AI token, you are guessing — and a guess dressed as a configuration is worse than no configuration, because it stops you measuring.

What happens if your rate limiter hits /robots.txt?

Your whole site goes dark for compliant crawlers, which is the most expensive mistake on this page. RFC 9309 is unambiguous about server errors: if the robots.txt file "is unreachable due to server or network errors," then "the crawler MUST assume complete disallow." A rate limiter, a WAF challenge, or a maintenance page returning 503 on /robots.txt therefore reads as disallow everything, not as slow down.

Two details make it worse. The 4xx case behaves oppositely — if robots.txt is unavailable with a client error, "the crawler MAY access any resources on the server" — so the same limiter can produce opposite outcomes depending on the status code it emits. And caching extends the damage: crawlers "SHOULD NOT use the cached version for more than 24 hours," which means a bad robots.txt response can govern a crawler's behavior for a full day after you have fixed the underlying problem.

The rule that follows is short. Exempt /robots.txt and your sitemap files from every rate limit, bot challenge and maintenance mode you operate, and check it after every CDN change. This is also the reason to know exactly what your edge serves for that path — the audit for that lives in Cloudflare's AI crawler controls.

What can you actually configure at the edge?

Blocking, mostly — the controls that exist are binary rather than graduated. Cloudflare's July 2025 content-controls post documents a managed robots.txt that maintains AI-bot directives for you, an option to block AI bots "only on portions of their site that are monetized through ads," and the earlier zone-wide block, with all of it available "to all Cloudflare customers, including everyone on a Free plan." What that post does not document is an AI-crawler throttle.

So if you reach for a generic rate-limiting rule at your CDN, understand what it is: pattern-based limiting that is not crawler-aware, applying the same counter to a documented AI crawler, an ordinary scraper and a burst of real users behind one NAT. It will work in the sense that requests get refused. Whether the refusal costs you retrieval is undefined, per the table above.

There is also a category robots-based throttling cannot touch at all. OpenAI documents that for ChatGPT-User, "because these actions are initiated by a user, robots.txt rules may not apply," and Perplexity documents that Perplexity-User "generally ignores robots.txt rules" for the same reason. A user asking an assistant to open your page is closer to a browser visit than a crawl, and any policy you write should treat it that way.

When is rate limiting the wrong answer?

Whenever the real problem is the meter, not the traffic — which on most stacks it is. The design rule from our car-leasing build, which runs a metered market-data API under a heavily crawled deals surface: cache markets whole, refresh them on a schedule priced against the vendor's per-call rates, and serve every visitor — human or bot — from the cache. Usage then scales with markets times refresh cadence, never with visitors, so a crawler IP storm cannot multiply the bill [our data].

The general form is boring and durable: static-render what can be static, cache whole datasets instead of querying per request, and let scheduled jobs be the only thing that touches anything metered. Do that and the cost side of the argument collapses, at which point crawler policy becomes a pure strategy question rather than a budget one. The dollar version of that argument, including what allowing crawlers actually costs, is in what blocking AI crawlers costs and saves; the business-model decision itself belongs to should publishers block or allow AI crawlers.

Against our own interest as a shop whose sites want to be retrieved: if you are a pageview-funded publisher, the honest recommendation is not to throttle at all. Decide, block cleanly with enforcement, and skip the middle option — a half-throttle gives you undefined behavior on the retrieval side and unclear savings on the cost side.

How do you know a limit isn't already in place?

Test per user agent, before you assume your configuration is the only one acting. On our auto-finance migration we found a legacy origin returning 403 to GPTBot specifically while its robots.txt explicitly allowed it — a host-level bot setting nobody had reviewed, invisible from the file everyone was reading [our data]. Separating that from ordinary rate-limit noise took interleaved fetches spaced about 70 seconds apart.

Pace matters more than tooling here. Space probes about 60 seconds apart, make one pass, and never re-test from an IP that has already been hammered — a burst of unpaced requests can get your own machine challenged, after which every later result looks like data and is not. The full verification procedure, including how to confirm a bot's identity before believing anything it did, is in verifying real vs fake AI bots, and the reason any of this is worth protecting is the subject of our generative engine optimization guide.