X-Robots-Tag

The X-Robots-Tag is an HTTP response header that carries the same indexing and serving rules as the robots meta tag. Google's specification states it plainly: "Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag."

The rules are identical. What differs is the transport, and the transport is the whole reason this term needs its own entry: instructions delivered in a header are set by infrastructure, not by whoever writes the page.

What can an X-Robots-Tag actually say?

Anything the robots meta tag can say, in the same vocabulary — noindex, nofollow, nosnippet, max-snippet, and the rest. Google's example of the header in a response is a single line alongside the status code:

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)

Three further mechanics are documented on the same page. The header may optionally name a user agent before the rule, in the form X-Robots-Tag: googlebot: nofollow, which scopes the instruction to one crawler. Multiple headers can be combined in one response, or several rules given as a comma-separated list. And because a meta tag cannot exist inside an image or a PDF, the header is the only way to apply robots rules to non-HTML files — Google publishes Apache and NGINX configuration examples for that case specifically (Google, robots meta tag and X-Robots-Tag specifications).

Why is a header more dangerous than a meta tag?

Because nobody looks at it. A meta tag is in the page source, where a developer, an auditor, or a curious client will eventually see it. A response header is invisible in the browser, invisible in view-source, and invisible in every content management interface — and it can be set by any layer between your application and the visitor:

The failure this produces is quiet by construction. The page renders, the content is correct, the markup is clean, and the page is simply gone from search — with no error, no alert, and nothing in the HTML to explain it.

One documented interaction makes the diagnosis harder. Google states that "if a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored." Blocking a URL does not deliver a noindex; it prevents the crawler from ever reading one.

Does an X-Robots-Tag change what AI surfaces can use?

Yes, through the same directives, because Google's AI features are documented as consumers of the snippet system. A noindex delivered by header removes the page from the index, and Google's AI features documentation ties AI Overview and AI Mode eligibility to pages being indexed and snippet-eligible — so one header removes classic results and AI surfaces at the same moment.

nosnippet and max-snippet behave the same way in a header as in a tag, including their documented AI effects. What each directive trades away, and why no AI-only opt-out exists, is worked through in the only granular AI Overview controls; the object all of them act on is defined in snippet. This page owns only the delivery mechanism.

How do you check for one?

Read the response headers, not the page. One request does it:

curl -sI https://example.com/page/ | grep -i x-robots-tag

Silence means no header on that response, which is the normal state. Check each environment separately — production, staging, and the URL as served through any CDN or proxy — because a header set at the edge will not appear when you request the origin directly, and a header set by a staging platform will not appear when you request production. Check after every infrastructure change, not only after content changes, since the header has nothing to do with content.

What do we do about it on our own builds?

Two layers, in-band and out-of-band. On our auto-finance rebuild the edge Worker deletes the x-robots-tag header from every response it proxies from the static origin, because the staging platform behind it sets that header by default; separately, an hourly external check requests the live site and watches for a stray noindex from any source [our data]. The routing architecture both live inside is described in adding a citable content layer to a live WordPress domain.

Be exact about what that record is. Both measures are precautions designed into the build. Nothing leaked, no page was deindexed, and this is not an incident report — the reason the defense exists is that the failure is invisible, not that we have watched it happen [our data].

Against our own interest: for a single-origin site with no CDN rules and no staging platform, this is a two-minute curl check once, not a monitoring project. The layer count is what creates the risk, and most sites do not have the layers. Where header-level controls sit in the wider visibility picture is mapped in our generative engine optimization guide.