How to Fact-Audit Content at Scale: The Refute-First Process

Fact-auditing at scale means auditing claims, not pages: every statistic, price, date, and legal assertion goes into a ledger, and each one gets checked by attempting to refute it from the exact source the page cites. This page is the operational process; the three-build record of what that process caught — with counts and dates — is documented separately in what fact-auditing 400+ pages catches.

The short version of why the posture matters: the worst error our audits ever caught was a fabricated state tax regulation, cited to a real source that says the opposite — and it read as the best-sourced sentence on its page [our data].

What is a refute-first fact audit?

A refute-first audit verifies each claim by trying to disprove it from the source the page itself cites. The auditor's question is never "does this sound right?" — it is "does this exact source actually say this?" Reading to confirm passes anything plausible; reading to refute is the only posture that catches a claim whose citation points at a source contradicting it.

The distinction is not academic. Plausibility-reading would have passed the fabricated regulation above, because the sentence was fluent, specific, and carried a real citation. Refutation caught it in one lookup [our data].

How do I build the claim ledger?

Extract every checkable assertion in the corpus into one table, one row per claim, before verifying anything. The ledger our audits commit carries 4 fields [our data]:

Two columns worth adding if you are building your own: a claim type (statistic, price, date, legal/regulatory, derived figure), which lets you batch verification work by kind, and the gate rule an error produced, which keeps the audit-to-gate loop visible in one artifact.

Two properties make the ledger work. It is exhaustive — a claim not in the ledger is a claim that never gets checked — and it is committed to the repository when done, which turns the audit from a private assurance into a checkable artifact.

Batch the rows by type before verifying, not by page. All derived figures go to the recompute script in one pass, all claims citing one source get checked in one reading of that source, and all legal-adjacent claims get the strictest reviewer. Type-batching is where the scale economics live — page-by-page auditing rereads the same sources dozens of times.

How do I verify each claim?

In three passes, cheapest first. Trace: confirm each claim's source exists and is the kind of source allowed to back it — on our builds that means an approved source list or a first-party record, and a claim with neither is already failed. Refute: open the source and try to disprove the claim from it, quoting the source's actual figure next to the page's. Recompute: every derived number — payments, percentages, deltas — is recalculated by script and compared against the page, never eyeballed.

The record shows why each pass earns its place. The refute pass on our greenfield insurance build checked 445 claims and corrected 40, including a statutory exemption published as "roughly $10,000" where the regulation says $12,997 — a hedge concealing a 30% error [our data]. The recompute pass on our auto-finance rebuild covered 461 payment figures and found 460 correct within $1; the single miss was a propagated error, wrong everywhere it had been reused [our data]. Hedges get no immunity: every "roughly" is treated as an unverified claim to verify or cut.

What do you do with what the audit finds?

Correct the pages, commit the ledger, and — the step that actually compounds — convert each error class into an automated build rule. A retired claim gains a build-failing regex the day it is retired, because our auto-finance audit found one retired claim had leaked back into 6 pages via writers faithfully matching stale exemplars [our data]. Derived figures are computed by script and used verbatim because of the propagated math error. Each audit hardens the gates the next tranche is written under.

That loop — audit finds, gate encodes — is the quality machinery inside the authority engine, and it is the honest answer to the reliability bar Google's people-first guidance describes: content that demonstrates its trustworthiness (Google, helpful-content guidance). Whether AI-assisted drafting itself is a search risk is a separate question with its own evidence, covered in does Google penalize AI content? — the audit is how AI-assisted pages earn the reliability the guidance asks for, whatever wrote the first draft.

Is a full audit always worth its cost?

Not always, and we sell audits — so weigh this against interest. The audit is the expensive phase of a content operation and it never gets cheaper with scale, because the unit of work is the claim and claims accumulate with pages. The full arithmetic — per-claim and per-page — is worked out in what fact-auditing costs.

The honest triage: pages dense with figures, tables, or legal-adjacent assertions get the full refute-first treatment first, because that is where our worst finds clustered [our data]. A corpus of genuinely figure-light pages can defensibly start with a targeted audit of its few checkable claims. What has no defensible version is skipping verification on a surface an answer engine may quote — an engine that lifts your wrong number republishes it under your name.