PerplexityBot and Perplexity-User: What They Do on Your Site

PerplexityBot in your server logs is doing one specific job, and the name in the log line does not tell you which of Perplexity's two agents you are actually looking at — or whether it is Perplexity at all. This page completes our per-crawler series alongside the GPTBot and ClaudeBot pages: what Perplexity documents about each agent, the measured crawl-to-click economics, what our own fleet's logs show, and the robots.txt reality most guides get wrong.

What is PerplexityBot?

PerplexityBot is the crawler that builds Perplexity's search index — the corpus its answers retrieve from. Per Perplexity's crawler documentation, it crawls the web for that index and respects robots.txt directives; it is a search indexer in the mold of a classic search crawler, not a foundation-model training bot.

The job description carries the stakes. If PerplexityBot cannot reach your pages, they cannot enter the index Perplexity's answers draw on — so this single token is the visibility gate for the whole platform. In our fleet's logs, PerplexityBot appears on all 3 production builds [our data]; we have not yet published dated per-bot request counts, and when we do they will be log extracts, not estimates.

What is Perplexity-User, and why does it ignore robots.txt?

Perplexity-User fetches a specific page because a person asked Perplexity something that requires it — and Perplexity states it generally ignores robots.txt rules, on the reasoning that a human requested the page. It is a live, on-demand fetcher: no browsing session, no index building, one page retrieved to serve one answer.

Per Perplexity's crawler documentation, current as of August 2026.

The posture is not unique — OpenAI says robots rules "may not apply" to ChatGPT-User — but it has a practical consequence most robots.txt guides skip: you cannot fully block Perplexity with robots.txt alone. The file, as defined in RFC 9309, is a compliance protocol, not an access control; it binds crawlers that choose to honor it. A disallow line stops the indexer. Stopping user-triggered fetches requires enforcement at the CDN or WAF layer, against Perplexity's published IPs. Where each agent sits among the full documented roster is in the complete AI crawler list.

How much does Perplexity crawl versus send back?

About 195 pages fetched per referral click, by Cloudflare's July 2025 measurement — and the trend is toward more extraction, not less. Cloudflare's crawl-to-click analysis put Perplexity's ratio at roughly 54:1 in January 2025 and near 195:1 by July, a more-than-3x increase in 6 months (Cloudflare, 2025). For scale: Google measured about 5.4:1 the same month, OpenAI ~1,091:1, Anthropic ~38,066:1. The metric and how to compute your own site's version is defined at crawl-to-refer ratio.

Our own instruments tell the same story in miniature: PerplexityBot fetches in the server logs on all 3 builds, and Perplexity referrals a small slice of the already-small AI-referral totals tagged on our oldest build [our data]. Plan on presence in answers, not sessions — and treat any pitch built on "Perplexity traffic" as ahead of the platform's measured economics.

How do you verify a Perplexity bot is real?

Check the source IP against the JSON files Perplexity publishes — the user-agent string alone proves nothing. Perplexity's crawler documentation links machine-readable IP lists for its agents, which makes verification a filter operation: a request claiming a Perplexity agent from an IP outside the published lists is an impersonator, and scrapers borrow famous bot names constantly.

This step is not optional if you plan to act on log data. Blocking decisions, rate limits, and any published claim about "what PerplexityBot does" on your site are only as good as the verification underneath them — an unverified log analysis measures whoever felt like wearing the name that week.

How do you block, limit, or allow the Perplexity agents?

One robots.txt group per decision, using the verbatim tokens from Perplexity's documentation:

# Remove your pages from Perplexity's search index
User-agent: PerplexityBot
Disallow: /

# Signal disallow to the user-triggered fetcher
# (Perplexity states this agent generally ignores robots.txt)
User-agent: Perplexity-User
Disallow: /

The first group is effective on its own terms: Perplexity documents PerplexityBot as respecting the file, and blocking it takes your pages out of the answer index. The second group is a signal, not a stop — enforcement against user-triggered fetches belongs to your CDN or WAF, keyed to the published IPs. There is no documented Crawl-delay support here, so the middle path Anthropic offers has no Perplexity equivalent; rate limiting, too, is an edge-layer job.

Should you block Perplexity?

Decide it by business model, not by mood — and either answer can be right. A pageview-monetized publisher looking at 195 fetches per returned click is subsidizing an answer engine that keeps the reader; blocking PerplexityBot is a coherent response, and the per-model logic is worked through in the robots.txt decision framework.

We take the opposite position for our own fleet: both agents allowed on all 3 builds, because our sites monetize being known and cited rather than the pageview itself [our data]. An index that can retrieve us and a fetcher that can read us live are distribution. If your goal is that side of the trade — being the source Perplexity's answers select — the visibility work is covered in how to get recommended by Perplexity and, platform-independently, in our generative engine optimization guide — and no robots.txt setting can promise the selection itself.