Bytespider: ByteDance's Undocumented AI Crawler

Bytespider is the AI crawler people block without being able to say what it does — and that is not the blocker's fault. ByteDance, the company behind TikTok, operates it without any of the documentation every other major AI crawler ships: no purpose statement, no robots.txt commitment, no IP list. This page collects what actually is known — Cloudflare's measurements, the RFC 9309 mechanics, and what we found on our own infrastructure — and is honest about the large remainder that is not.

What is Bytespider?

Bytespider is ByteDance's web crawler, and for a stretch of 2024 it was the single heaviest AI crawler on Cloudflare's network. Cloudflare's July 2025 content-controls update calls it their previous top bot, with its access reaching more than 40% of Cloudflare-protected websites before the decline began.

What it collects content for has never been officially stated. The standard assumption — training data for ByteDance's AI models — is consistent with its fetch-heavy, refer-nothing behavior in Cloudflare's data, but it remains an inference. No ByteDance page we can cite describes the bot's job, and we treat that absence as a fact about the bot rather than a gap to fill with guesses.

Why is there no official Bytespider documentation?

No public answer exists, and the absence is the single most important thing to know about this crawler. The five companies on the documented AI crawler roster — OpenAI, Anthropic, Perplexity, Google, Microsoft — each publish three things: what their bots do, whether they honor robots.txt, and how to verify them by IP. ByteDance publishes none of the three.

That asymmetry has a practical consequence for every operator decision on this page. You cannot weigh the cost of blocking a bot whose benefit is undocumented, you cannot hold it to a compliance policy it never made, and you cannot verify that a "Bytespider" log line is really ByteDance. Every downstream call becomes a default-deny-or-default-allow judgment instead of an informed trade.

How much does Bytespider actually crawl?

Much less than it did: every Cloudflare measurement of Bytespider points the same direction — steeply down between mid-2024 and mid-2025.

Why the collapse happened is not something Cloudflare's posts explain, and ByteDance has said nothing citable. What the numbers establish is narrower but useful: the era of Bytespider as the dominant AI crawler was over by mid-2025, so a block-Bytespider argument built on 2024 volume figures is arguing with a ghost.

Does Bytespider send any traffic back?

Almost none is documented, and no ByteDance product publishes a citation or referral mechanism for web sources. Cloudflare's crawl-to-click data measured ByteDance's crawl-to-refer ratio at roughly 18:1 in January 2025, falling to about 0.9:1 by July 2025 — a 95% decline that coincides with the crawl-volume collapse in the table above, so read it as less crawling rather than a surge of generosity.

Contrast that with the platforms that do document an exchange: OpenAI, Anthropic, and Perplexity all operate answer surfaces that cite sources, and Google's AI features link out. Whatever Bytespider's fetches feed, there is no documented path by which your pages earn a citation or a visitor from it. That fact does most of the work in the blocking decision below.

Should you block Bytespider?

If you intend to block any AI crawler, this is the cheapest one: no documented answer surface, search product, or referral path depends on it, so a block forfeits nothing any vendor has committed to. That is why it appears in block-by-default configurations — Cloudflare's managed robots.txt disallows it as 1 of 8 AI-crawler tokens, a default we encountered directly when a pre-cutover audit found that file being auto-served on a production zone nobody had configured [our data].

We allow it anyway, and the reasoning is worth stating because it cuts against the easy recommendation. Our robots.txt posture across all 3 production builds is allow-everything [our data]: our sites earn by being retrievable, we prefer one consistent posture over per-bot exceptions we would have to re-litigate quarterly, and Bytespider's collapsed volume makes its cost to us negligible. A pageview-monetized publisher, or any operator paying for bandwidth by the fetch, reaches the opposite conclusion with the same facts — the business-model arithmetic is worked through in should publishers block or allow AI crawlers.

How do you block Bytespider?

Write a standard RFC 9309 group — the syntax works for any token, documented or not:

User-agent: Bytespider
Disallow: /

Then treat that line as a request, not a result. RFC 9309 is explicit that robots.txt rules "are not a form of access authorization," and compliance is a commitment each operator makes — one ByteDance has never published for Bytespider. With no compliance policy and no published IP list to verify against, dependable blocking means CDN or WAF enforcement, which is exactly the layer where Cloudflare's default already handles it.

Slot the decision into your wider file using the robots.txt decision framework — Bytespider is one row in a per-token exercise, not a policy on its own. And if the reason you leave AI crawlers open is to earn citations from the engines that do send them, the work that makes those fetches worth anything is the subject of our generative engine optimization guide.