How to Structure a Content Library for AI Search

A content library for AI search is organized around questions, not topics: one question per URL, a small set of clusters that each do a distinctly different job, and a glossary carrying the definitions every other page leans on. Answer engines retrieve passages that answer a question, so the architecture's whole purpose is to make sure a passage exists for each question you want to be findable for — and that a crawler can reach it in raw HTML.

This page is the cluster-and-URL design step only. The full operating system around it — research, contract, gates, distribution, measurement — is the authority engine.

Why is cluster design a retrieval decision?

Because retrieval works at passage scale, and passage quality is decided by what a URL is about. A page built around one question opens with an answer to that question and can be quoted whole. A page built around a topic — "auto loans," "umbrella insurance" — contains many partial answers and no complete one, so every passage it offers competes weakly.

Cluster boundaries are the same decision one level up. When two clusters do overlapping jobs, writers place pages by intuition and the same question ends up answered in two places with two different answers. The cluster list is therefore a short list of jobs, written down before any page exists.

What are the five clusters, and what does each one do?

Five jobs, five clusters. Pillars carry the system-level explanation a whole vertical hangs off. Learn answers how and why questions. Cost answers how much, with the arithmetic shown. Scenarios are dated field reports — what happened, what changed, what generalizes. Glossary defines the vocabulary everything else uses.

The test for adding a sixth cluster is whether its pages would have a genuinely different job, schema type, and question shape. One of our builds added a tools cluster on that basis; nothing else has cleared it.

How many pages does each cluster need?

As many as the question inventory supports, which varies enormously by niche. The counts below are our three production builds' architectures — the first measured as published pages at audit time, the second and third as planned URL counts against which files were still landing [our data]:

One caveat on the middle column before the reading: its cluster counts sum to 381 against a headline planned total of ~365. We print the figure our architecture doc recorded rather than a tidied one, and the 16-page gap is planning drift we have not gone back to reconcile [our data].

Three things are worth reading out of that table. The ratios differ by niche, not by preference: auto-finance carries a heavy glossary because its vocabulary is inconsistently defined across the web, while the insurance build carries heavy scenarios because its buyers search in situations. Cost is large only where the niche has genuinely distinct cost questions — provinces, states, policy types — and it is the cluster most likely to inflate into thin pages if nobody gates it. And the shape held across all three: same five clusters, three verticals, two countries [our data].

At last commit those architectures held 188, 298 and 106 content files respectively [our data]. What a library of that size costs to produce is priced in the 100–200 page library budget.

What belongs in the glossary?

Every term the rest of the library uses without explaining. The glossary is the layer that lets other pages stay on their own question: a cost page can say "crawl-to-refer ratio" and link down instead of spending a paragraph defining it, which keeps its own passages tight.

Two rules make the glossary carry that weight. Each entry opens with a standalone definition of about one sentence — the extractable unit — and then explains what the term changes for the reader. And terms are prioritized by how inconsistently they are already answered: on our auto-finance rebuild, about 35 of 124 planned terms were flagged as answered inconsistently by AI assistants and written first [our data]. A definitional page is at its most useful where the existing answers disagree with each other.

Where should perishable numbers live?

In a separate data layer, never hard-coded into static pages. Our leasing build made this explicit: a live offers board carried every expiring figure with verification and expiry stamps, while the 102-page knowledge layer explained mechanics and linked the board rather than restating its numbers [our data]. The board went live 26 days before the knowledge layer existed.

The architectural rule generalizes past deals sites. Any figure with a shelf life — prices, rates, tool pricing, platform limits — either gets a date stamp and a review cadence, or it belongs in a layer that regenerates itself. A static page that quotes a number it cannot maintain becomes wrong quietly, and a library full of quietly wrong numbers is the opposite of a citable one.

In what order should the library get built?

In explicit priority order, decided before writing starts. Rank every planned URL: definitional terms the web answers inconsistently, then the questions with demonstrated demand, then the long tail. Where a cluster expands into templated variants, hold the rollout behind an indexation gate — our auto-finance architecture put 64 cost pages behind exactly that check on its state rollout [our data].

Two things get decided in the same planning pass. Inbound links per page, because alphabetical sibling linking is what produces orphans at scale — the failure mode and its fix are in internal linking for a content library. And the tranche assignment, so parallel writers work from one binding brief instead of improvising: that machinery is programmatic page production.

What does architecture not fix?

Three things, plainly. It does not make thin pages worth publishing — a templated variant with nothing specific to say is thin whether or not it sits in a well-named cluster, and Google's Search Essentials bar applies to it exactly as it does to everything else. It does not decide whether a question deserves a page; demand research does. And it cannot produce citations, rankings or traffic — no structure can, and anyone selling one that does is selling something else.

Against our own interest, since a five-cluster architecture is what we build: a site with twenty pages does not need this. Twenty pages need one clear question each and a navigation menu. The architecture starts paying when the page count passes the point where a person can hold the whole library in their head — the scale at which a library becomes an authority surface, described end to end in our generative engine optimization guide.