Field report
When a Platform's Own AI Fabricated Coverage Claims
The short answer
What happens when a platform's own AI suggests copy for you?
On our insurance build, a publishing platform's own content-suggestion feature proposed copy that fabricated coverage claims — blanket statements about what a policy category can never cover, false and against controlling case law. Because the copy carried no figure, price or date, 0 of our automated gates could key on it. The rule since: take the topic signal, never the sentences [our data].
Most warnings about AI-written content assume the AI is yours. This one was not. On our insurance build, the fabricated claims arrived inside a publishing platform's own content-suggestion feature — a helper offered by a product we already trusted, proposing copy for a brand in a YMYL vertical [our data].
The claims were blanket statements about what a policy category can never cover. They were false, and they ran against controlling case law. What our record does not say is how that was noticed: it documents the fabrication and the rule that followed it, not a reviewer, a check, or a stage in the process. We are not going to invent one. The platform is not named here either, because the operator has not cleared naming it, and because the failure is not specific to one product.
What was the situation?
We use platform-side content suggestions as demand research on our insurance build [our data]. The feature reports what people are actually searching for inside that platform, which is a cheap and genuinely useful input when you are deciding what to publish next — it is an observation about behavior, not a claim about the world.
Alongside the topics, the feature offered copy. That is the part that failed. A suggestion tool that surfaces a question is doing something different in kind from a suggestion tool that answers it, and the interface presented both as the same feature.
What did the platform's AI actually suggest?
Coverage assertions that were simply not true — categorical "no policy of this type can cover…" framing, in a vertical where that statement has a legal answer and the legal answer went the other way [our data]. The copy was fluent, confident, and formatted like every other suggestion in the panel.
Four properties of the failure are worth separating, because they generalize past this incident.
| Property of the failure | Why it mattered |
|---|---|
| It contained no figure, price or date | Every numeric gate we run had nothing to key on |
| It was categorical, not hedged | Categorical claims are the most quotable and the most falsifiable |
| It appeared inside a trusted product | Implied endorsement lowers the scrutiny a draft receives |
| The topic was YMYL | A wrong answer here is a consumer-harm problem before it is a content problem |
Google's people-first guidance frames the publishing bar in terms of content that demonstrates reliability and first-hand expertise (Google, helpful content guidance). A fabricated coverage rule fails that bar regardless of who typed it — and there is a second reason to care about the sentence rather than the source. Google's AI features documentation ties AI Overview and AI Mode eligibility to pages being indexed and eligible to appear with a snippet, which is another way of saying that anything you publish is material an answer engine may draw on. A false sentence does not sit quietly on your page. It becomes quotable.
Why is the demand signal safe when the sentences are not?
Because they are different types of statement. "People search for this" is a report about observed behavior inside a platform; "no policy of this type covers that" is an assertion about the world that has to be true. The first can be wrong and cost you a wasted page. The second can be wrong and cost you a reader's decision.
| What the feature offered | What we do with it | Why |
|---|---|---|
| Topics with real search demand | Use as-is, as an authoring queue | A demand observation, not a claim |
| Suggested titles and hooks | Rewrite from scratch | Phrasing smuggles in implied claims |
| Suggested body copy | Discard | It fabricated coverage claims, which is enough [our data] |
This is the whole rule, and it is smaller than the incident suggests: take the signal, never the sentences.
What did we change?
We stopped letting any generator author claims. On our insurance build, video scripts are generated from an already-audited source page and constrained to restate what that page says, rather than answering from the model's own knowledge [our data]. That is the structural fix — correctness is inherited from an audit that already happened, instead of being re-established every run by a reviewer who may be tired.
On our leasing build the same idea runs as code rather than instruction: the video factory has a fact gate that aborts a render when a dollar figure in the generated script is not present on the live offer board [our data]. A prompt asks; a gate refuses.
The related and more general argument — that compliance language belongs in code rather than in a prompt, and how contextual gating avoids blocking your own correction pages — is worked through in YMYL niches and AI Overviews, which owns that reasoning. What the automated video pipeline costs to run is a separate question, answered in video automation cost.
What does this incident not fix?
It does not fix prose. The gate on our leasing build catches a wrong number; it cannot catch a wrong sentence. The copy that started this page contained nothing a regular expression could match, and we should be honest that no system we run would have found it. Our record does not document what did — which is its own lesson, because a catch you cannot name is a catch you cannot schedule [our data].
That limit points at the uncomfortable symmetry. Our own pipeline is AI-assisted too, and it produces the same class of confident error, which is why every build carries a human audit stage with a committed ledger; the method is described in our fact-audit process, and the wider question of whether AI-assisted production is itself a liability is handled in does Google penalize AI content. We did not discover that platform suggestion tools are uniquely careless. We discovered that we had been applying a lower standard to text that arrived inside a product than to text our own pipeline generated.
Against our own interest: this is one incident on one build, not a measured error rate for anything. We did not count the fabricated sentences and our record does not either, we have not sampled the feature systematically, and one incident is a reason to add a review step rather than evidence about a vendor. There is also no traffic, ranking, or revenue outcome attached to this record — the reportable result is a rule that replaced a judgment call.
What is the generalizable rule?
Separate the demand signal from the sentences, and let every generated claim inherit its correctness from something already verified. Any AI that proposes copy — yours, a vendor's, or one built into a platform you pay for — is a drafting tool, and a drafting tool's output is a draft no matter how officially it is presented.
Three checks fall out of it. Ask what type of statement you are being handed: an observation about demand, or an assertion about the world. Point every generator at an audited source instead of at the open model, so it restates rather than invents. And enforce in code whatever is non-negotiable — disclaimers, figures, jurisdiction notes — because those are the rules a deadline erodes first.
None of that makes a library citable on its own. It makes a library defensible, which is the prerequisite: an answer engine that quotes you republishes your claims under your brand's name, and that is the whole reason accuracy is a distribution concern and not just an editorial one — the argument developed in our guide to generative engine optimization.
Frequently asked questions
What happens when a platform's own AI suggests copy for you?
It can be wrong in ways that look authoritative. On our insurance build, a publishing platform's suggestion feature proposed coverage claims that were false and contradicted controlling case law. Our record documents the fabrication and the rule we adopted afterwards; it does not document who noticed, how, or at which stage, so we will not attribute the catch to any particular check [our data].
Why is AI-suggested copy riskier than a normal AI draft?
Because of where it appears. Copy offered inside a product you already trust arrives with implied endorsement, so it invites less scrutiny than the same sentence pasted from a chatbot. The suggestion is a draft either way, and it deserves the same audit [our data].
Would a fact-checking gate have caught it?
No. The copy carried no statistic, no price and no date, and every automated check we run keys on figures, sources or retired phrasings. A false claim written in plain prose passes all of them, which is why a human read stays in the process [our data].
Should you stop using platform content suggestions?
We did not. The demand signal is genuinely useful — it reports what people search for inside that platform. The rule we adopted separates the two halves: use the topics, rewrite every sentence, and never let suggested copy introduce a claim [our data].
What does it mean to fact-lock a generation prompt?
It means the generator is pointed at one already-audited source page and permitted to restate what that page says, rather than answering from the model's own knowledge. Correctness is inherited from the audit that page already passed, instead of being re-established every run [our data].