The layers between a question and a fact

I use one research engine for decisions across my personal life, my day job and the sauna directory I run. What it gets used for has its own page. This one is about how it decides what is true.

It started with a mistake. In July I ran one research question through three AI tools, and two of them gave a €2.75bn figure to the wrong person. Most of the tools agreed, and they were wrong. So the engine is built on one rule: how many models say something never decides whether it is true. The source decides. Everything else is layers that make that rule hold.

Every layer catches the one before followed one early run from research through drafting to editorial review. This piece stays on the research side and goes layer by layer, including the ones added since.

A brief I approve

A rough question becomes a written research brief: what to cover, which sources count, what to keep separate. Nothing runs until I have read and edited it. This is where mistakes are cheapest. Early on, the engine's own name in a brief's title leaked into one model's searches and filled its sources with irrelevant pages; tool names now stay out of briefs.

Prices get extra rules. On my first run comparing cars to buy, I pasted in a link to a listing and the recommendation changed. A second link within the hour changed it again; the models had read old prices from out-of-date tables. Now the brief asks me for any links I already have, each model must give the web address and the date it saw every price, and anything behind a quote form is named as a gap rather than estimated.

Four models, working blind

The brief goes to Claude research agents, Perplexity, OpenAI and Gemini, each searching the web its own way without seeing the others' work. The independence has to be real. Perplexity's newer options run on OpenAI's models, which would have quietly made it a second OpenAI. It is held to Perplexity's own model, and if that model is withdrawn the leg is switched off rather than swapped.

More models buy reach, not proof. I added Gemini expecting it to find Asia-Pacific sources the others missed, and its first run disproved that: Perplexity found them without help. What Gemini did bring was official sources. It was the only model to reach stock-exchange filings and a government website, with 20 distinct sources to the other two external models' 11 and 8.

A ledger of single claims

The four answers are broken into single claims, each with the models that made it and the sources they cite. Agreement is recorded, and it is weak evidence; models learn from overlapping material and cite each other. The claims that go forward are the ones where the answer to "would the conclusion change if this were false?" is yes. Numbers, names, dates and attributions come first.

Verification against the source

Each of those claims goes to a checking agent whose instructions are to break it, not confirm it. It opens every cited page itself, because a citation is a lead and not evidence. It looks for a second source that did not come from the first; two articles rewriting the same press release count as one. It never checks a claim against the output that produced it. A failed fetch is recorded as a tool failure, never as "not supported". Each claim leaves with a grade: verified, partly verified, wrong, or unconfirmed.

Two of its rules came from my own mistakes. "Wrong" needs evidence that contradicts the claim. On an early run, claims were marked wrong because nothing supported them, and they turned out true once an archived original surfaced. Failing to find support makes a claim unconfirmed, which keeps it as an open question instead of deleting it. And a page that comes back blank is a tooling problem, not an evidence limit: two facts from a regulator sat at "partly verified" through two weekly reviews and became verified in minutes once someone opened them in a browser.

This is the layer that does most of the catching. On a car lease decision in September, one model returned prices, a company registration number, review counts and energy tariff rates, every one marked "confirmed" and every one wrong. Verification killed them all.

Lists of firms and people get a sister layer. Each row is graded twice: how well the fact is evidenced, and what has changed since the last record. The two stay separate on purpose. A person marked as having moved jobs makes any work email on that row stale, whatever grade the move itself carries.

My market-intelligence pipeline goes one step further on a single point: every fact must carry a quote that software matches against the source, character for character. That rule has its own write-up.

Checks on the record

Every verification record goes through fifteen automatic checks on its shape, and passing means all fifteen. When I first pointed them at real records, they found four bugs in themselves before they found anything in the records. Then they found a real one: a record whose summary claimed 18 verified claims while its own table showed 25.

The findings document gets its own checks, and two of them exist because the engine misreported its own runs. For a month, the instructions stated which models were switched on, and that line was wrong four times. One run went out on a single model while the record said the others were unavailable; they had been connected for eleven days. The line was deleted, and the answer now comes only from the settings file and a live test before every run. A check now fails any findings document that presents itself as complete while its own record shows a model dropped out.

The second: a trial run skipped verification entirely and still passed nine checks out of nine. A shape check can only see what is on the page, and a stage that never ran leaves nothing there to fail. So a document that marks any claim verified must now link the verification record, and that record must exist and pass.

A second reading, blind

None of that says whether a "verified" deserved the word. So a sample of graded claims, weighted toward the ones marked verified and the ones marked wrong, goes to a fresh checker that sees only the claims, not the earlier grades or evidence. The first trial was five claims: four agreed, one differed by a single grade, none flatly contradicted. Any flat contradiction is re-opened and decided by reading both pieces of evidence.

A scorecard for each model

The last layer asks whether each model earns its place. It joins every graded claim to the models that made it. As of today, 294 of 529 graded claims across five runs can be traced to a model; the rest come from runs that did not record who said what, which the engine now requires. On those, about three quarters of OpenAI's claims were verified and under half of Gemini's. Of the 53 claims only Gemini made, 11% were verified and 45% were wrong or unconfirmed.

I haven't dropped Gemini. Its solo claims are mostly noise and its reach into official sources is real, and its output is already treated as leads, never findings. The rule stays at all four models on every run until the count is big enough to carry the decision.

Findings, then a decision

Only then does the engine write anything. In one mode it settles on the best-supported answer, and an outlier from one model is a suspect to verify or drop; in the other it maps the whole spread and keeps outliers as signal. If gaps remain, it can go back for those alone, at most twice. Every findings document opens with a summary of about five lines, and carries a section on conflicts and how they were settled, and another on what no source could confirm. Then I decide.

What the layers can't see

Two gaps are worth naming. The checks can only grade claims that were pulled out of the answers; a fact no model mentioned leaves nothing for any layer to catch. And my meeting-preparation tool is graded against a fixed set of saved test cases, but the research records aren't yet; the blind second reading stands in until they are. Both gaps are printed by the checks themselves, on every run, so a clean result can't be read as more than it is.