llms.txt: What It Is and How to Set It Up

llms.txt is a proposed convention for publishing a curated, Markdown-formatted map of your site's most important content at a fixed URL. First things first: this is the llms.txt proposal (llmstxt.org, 2024) — not an adopted standard, and no major AI engine has committed to reading it. This page walks through the format and a 30-minute setup, then shows what our own server logs say about who actually fetches the file.

What is llms.txt supposed to do?

The proposal's idea is a bandwidth-friendly front door for AI systems. Context windows are finite and HTML pages are heavy with navigation and scripts, so llms.txt offers a single plain-text file — served at /llms.txt — that names the site, summarizes it, and links its key pages with one-line descriptions. An assistant that honored it could orient itself from one small fetch.

The proposal also sketches a companion file, llms-full.txt, carrying full page content as Markdown, and suggests Markdown versions of individual pages. All of it is opt-in on both sides: sites may publish, engines may read. As of August 2026, publishing happened at meaningful scale; committed reading, as far as any public documentation or our logs show, did not.

How do you write an llms.txt file?

The format is deliberately simple: one H1, one blockquote, then H2 sections of annotated links. Here is a trimmed version of the shape we deploy:

# Magnitude Authority Engine

> A library on AI answer engines and generative engine optimization,
> written from three production authority builds.

Guides

• The Complete GEO Guide: every tactic tagged by evidence tier
• GEO vs SEO: what genuinely differs and what doesn't

Cost

• What an authority build costs: real production economics

Each element has a job, per the [proposal](https://llmstxt.org/):

| Element | Required by the proposal | Content |
|---|---|---|
| H1 title | Yes | Site or project name |
| Blockquote | Recommended | One-paragraph summary of what the site is |
| H2 sections | Optional | Groupings such as Docs, Guides, Pricing |
| Link lists | Optional | `[Title](URL): one-line description` per entry |
| `## Optional` section | Optional | Links an agent may skip when context is tight |

The ordered steps in the box above take you from inventory to verification;
the whole process fits in about 30 minutes for a typical site, and under 5 if
your build system generates the file from existing metadata, as ours does
across all 3 builds [our data].

## Should you bother deploying llms.txt?

The evidence says: only as a near-zero-cost bet, never as a tactic you expect
to move anything. The published debate is short. Google's John Mueller
compared llms.txt to the keywords meta tag
([Search Engine Journal](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/)) —
a signal sites volunteer about themselves that engines have little reason to
trust. [Search Engine Land's counterpoint](https://searchengineland.com/no-llms-txt-is-not-the-new-meta-keywords-458199)
argues the comparison is unfair: unlike the keywords tag, llms.txt carries no
spam incentive structure yet, costs almost nothing, and could become useful if
adoption arrives.

Our contribution is logs rather than opinion. We serve llms.txt on 3
production sites — an insurance lead-gen brand, an auto-finance authority
rebuild, and our own network — and log every request to the file: 0 requests
for `/llms.txt` from GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, or
PerplexityBot to date, on any of the 3 sites, while the same logs show AI
crawlers fetching HTML pages [our data]. The running 90-day watch, site by
site, is in
[our llms.txt log](/scenarios/llms-txt-90-days-three-sites/).

So both sides of the argument are right about different things. The skeptics
are right that nothing reads it today; the defenders are right that 30
minutes is a fair price for optionality. What is not defensible is paying for
llms.txt generation tools or services — and several are sold — on visibility
claims no log we have seen supports.

## How do you know if anything ever reads your llms.txt?

Your server or CDN logs are the only honest scoreboard. Filter requests by
path for `/llms.txt` and group them by user agent; on a CDN-fronted site the
same view is one filter in the analytics dashboard. We keep this logging
permanent on all 3 of our builds, so adoption — if it ever arrives — will
show up in our own data before it shows up in anyone's marketing [our data].

Interpret what you find against documented crawler names. OpenAI, Anthropic,
and Perplexity all publish their user-agent strings, so a genuine engine
fetch is identifiable the day it happens. Expect the ambient noise instead:
SEO audit tools checking whether the file exists, generic scrapers, the
occasional curious human. So far, that noise has been the file's
entire audience on our sites [our data]. A quarterly log check is plenty — a
file nobody fetches does not need a dashboard.

## What should you do instead of relying on llms.txt?

Spend your effort on the files and behaviors engines demonstrably use. AI
crawlers fetch HTML pages daily in our fleet's logs [our data], so the
highest-leverage work is making those pages extractable: direct answers in
raw HTML, self-contained passages, sourced claims — the practices cataloged
in our [complete GEO guide](/generative-engine-optimization/).

Access control belongs in robots.txt, the adopted standard every major
crawler documents compliance with — see
[the robots.txt decision framework for AI](/learn/robots-txt-for-ai-decision-framework/)
for per-bot choices, and
[the complete AI crawler list](/learn/ai-crawlers-complete-list/) for who is
actually fetching. If you deploy llms.txt too, do it with the setup above,
watch your own logs, and let the file earn attention only if the engines ever
give it any.