The 2,895-Line Robots.txt We Never Wrote

We audit before cutovers because the checklist says to, and most checks find nothing. This one found a 2,895-line robots.txt we never wrote, serving Disallow: / to 8 AI crawlers, on a domain scheduled to flip its apex to our stack the next day — while the robots.txt in git, the one every review had approved, said allow everything [our data]. This page is the field report: what we walked into, what the audit surfaced, what changed, and the rule that now runs on every build.

What was the situation?

A live-domain migration — our auto-finance authority rebuild — one day from its cutover, with an explicitly allow-everything crawl posture as strategy. The build's entire premise was the citation layer: hundreds of fact-audited pages meant to be fetched, retrieved, and quoted by answer engines, which is why its robots.txt contained no AI Disallow at all. The domain sat behind Cloudflare, and the migration design routed traffic through a Cloudflare Worker, so the proxy was about to become the front door for everything.

The stakes of a robots misfire were not hypothetical. Blocking-by-accident is the one failure that produces no error anywhere: crawlers simply stop coming, and the crawler list you expect in your logs quietly never appears.

What did the audit observe?

The scheduled pre-cutover audit on 2026-08-05 fetched /robots.txt from the outside and got back a file we did not recognize: 2,895 lines, opening with Cloudflare's managed directives, carrying Disallow: / for 8 AI crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, Amazonbot, and meta-externalagent [our data]. The zone's "Managed robots.txt" feature was active, and nobody on the project had enabled it.

Audit record, our auto-finance authority rebuild, 2026-08-05 [our data].

The mechanism made the divergence invisible by design: Cloudflare's feature answers the /robots.txt request at the edge and prepends its directives to the origin file (Cloudflare, July 2025) — the source file is never modified, so every internal check of the repo showed a clean allow posture. The third row was the audit's second find: the legacy origin was returning 403 to GPTBot specifically, robots.txt notwithstanding — a separate, server-level block that only surfaced through interleaved per-user-agent fetch tests spaced about 70 seconds apart to rule out rate limiting [our data].

And the timing detail that made it a near-miss rather than an incident: CDN settings act only on proxied traffic. The managed file was mostly harmless while the apex still pointed at the legacy host — and would have become the whole site's crawl policy the moment the flip went through.

What did we change?

Three things, the same day. We turned the managed robots.txt off in the zone's AI Crawl Control, restoring the repo file as the single served truth. We re-verified from the outside the way the audit found the problem: fetches of /robots.txt with a plain UA and then crawler UAs — GPTBot, ClaudeBot, CCBot, PerplexityBot, OAI-SearchBot — spaced about 60 seconds apart, one pass, each response compared byte-identical against git [our data].

Then we walked the rest of the zone's bot machinery — per-crawler block toggles, Bot Fight Mode — confirming every state matched a decision our strategy had actually made. That walk is now a standing checklist with its own page: what Cloudflare's AI controls actually do.

What was the outcome?

The cutover proceeded on schedule the next day with the correct file live, and the divergence never touched production traffic [our data]. No AI crawler saw the Disallow wall; the build's citation strategy went into its migration intact. The 403-to-GPTBot finding on the legacy origin was logged as its own defect in the same audit ledger — standing proof that a server can contradict its own robots.txt [our data].

What the incident cost was one audit afternoon. What it would have cost undetected is harder to bound: an allow-everything strategy serving block-everything to 8 crawlers, with no error message, no dashboard alert, and discovery only whenever someone eventually asked why the server logs showed no AI fetches. Silent misconfigurations are priced by their detection delay, and this one's detection path was a calendar entry.

What is the rule?

The robots.txt you wrote is a claim; the robots.txt that is served is the fact — and once a CDN proxies your traffic, those are different files until proven identical. Concretely, on every build we operate: audit the CDN's crawl-control state before any milestone that moves traffic through the proxy, verify the served file per crawler user agent against source, and treat every bot-related dashboard default as unreviewed strategy until someone maps it to a decision [our data].

To be fair to the vendor: Cloudflare's managed robots.txt is a legitimate, well-made control, and its economics case is real — Cloudflare's own measurements had training crawlers fetching tens of thousands of pages per referral in 2025, and a publisher who wants that refused gets professional-grade blocking free. The trap is not the feature. The trap is state without intent — which is a general property of infrastructure, and the reason "verify what is served" now sits one line above "publish something worth serving" in our generative engine optimization playbook.