The Number Nobody Measured

The statistic you are about to cite was probably never measured. It was written down, then repeated, and repetition is the only thing your search tool can see.

Here is the shape of it. Somebody needs a number for a blog post. They find a page titled "47 Remote Work Statistics for 2026." That page cites a second page. The second page cites a third. Somewhere around the fourth hop the trail stops at a page that cites nothing at all, because that is where the number was invented. Every link in the chain looks like sourcing. None of it is.

I spent a lot of the early 2000s building a challenge-response spam filter, back when that still looked like a winning idea. I learned one thing there that has stayed useful: when the check and the attack travel down the same channel, the check does not work. You cannot verify a message using the medium the message arrived on. That is exactly the position anyone doing research with a search tool is in right now.

Search measures agreement, not truth

This is the part that gets people, and it got me. Grounded search feels like verification. You ask whether a claim is supported, the tool goes and looks, and it comes back with agreement and a link. The loop feels closed.

It is not closed. Search does not measure whether a number is true. It measures whether the number is repeated. Those two things come apart precisely when a figure has been laundered through enough pages, because laundering is repetition. The more thoroughly a statistic has been copied around, the more confidently a search-grounded check will confirm it.

So the failure mode is not that verification is hard. It is that verification returns a clean result with the wrong sign. A number with no underlying study is not merely unverified. It is easier to verify than a real one, because the real one might only exist in a single PDF that nobody reblogged.

The machines are confidently wrong

The Tow Center for Digital Journalism ran the experiment properly. Klaudia Jaźwińska and Aisvarya Chandrasekar tested eight AI search products across 1,600 queries. Each one got a quote from a real article and was asked for the title, publisher, date and URL. Per their write-up in Columbia Journalism Review, the tools failed to retrieve the correct source more than 60% of the time. Perplexity, which markets itself specifically as a research tool, was the best of them and still got it wrong 37% of the time.

The number I keep coming back to is a smaller one. As Nieman Lab summarized it, of 134 incorrect citations ChatGPT produced, it hedged its language in 15. The other 119 were delivered straight.

That is the whole problem in one ratio. A tool that is wrong and sounds uncertain is a tool you check. A tool that is wrong and sounds certain is a tool you quote. It is the same structural complaint I have made about models having no intent. The performance of confidence is not evidence of correctness, and nothing in the training rewards the difference.

Citogenesis is not new, only faster

Randall Munroe named this in 2011, in xkcd 978. An unsourced claim goes into a wiki, a journalist repeats it, the wiki then cites the journalist, and the loop is sealed. He called it citogenesis and the term stuck because everyone recognized it immediately.

It kept happening. Slate documented the pattern in 2019. One case involved a wristwatch whose release year was wrong on Wikipedia and then repeated by the BBC. The error was afterwards defended using the BBC article, against primary sources that said otherwise. The correction was harder than the error.

What changed is throughput. Citogenesis used to need a journalist, a deadline, and a few months. Now a content pipeline can mint a hundred statistics pages in an afternoon, and a research agent can cite all of them before lunch. The mechanism is identical. The clock speed is not.

Follow the money to the roundup

Nobody writes "47 Remote Work Statistics for 2026" because they care about remote work. They write it because a page full of numbers ranks, and a page that ranks carries ads or a product link.

Look at the unit economics from the publisher's side and the behavior stops being mysterious. Running an actual survey costs real money and produces one number. Assembling forty numbers other people published costs an afternoon and produces forty chances to rank. There is no point in the funnel where accuracy pays and several where volume does. The incentive does not have to be malicious to be reliable, and reliable incentives beat good intentions every time.

This is the same engine behind every vendor benchmark I have ever had to take apart. As with the numbers vendors put in their decks, the question is never whether someone is lying. It is who paid for the measurement, and what they needed it to say.

What checking actually looks like

The useful move is to stop asking whether a claim is supported and start asking whether the cited page exists and did the work. Both are cheap to check and neither requires a model.

A dead URL is not automatically a fabricated one. Link rot is ordinary. The distinction that matters is whether the page was ever there, and the Wayback Machine will tell you for free:

# Was this URL ever archived? Empty [] means no snapshot, ever.
curl -s "https://web.archive.org/cdx/search/cdx?url=example.com/some/article&output=json&limit=5"

# Does it resolve for a browser? Many publishers 403 a bare curl.
curl -sSIL -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 \
  (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
  "https://example.com/some/article" | head -1

On a densely archived publisher, a URL with no snapshot at all is a strong signal the page never existed. On somebody's personal blog it signals nothing, because the crawler may simply never have gone there. Keep those two cases apart or you will start accusing people of inventing sources they actually wrote.

The second check is shape. Big publishers mint URLs to a fixed template. A wrong-shaped path is often a fabrication you can catch without a network round trip:

import re

# Harvard Business Review articles are /YYYY/MM/slug
HBR = re.compile(r"^/\d{4}/\d{2}/[a-z0-9-]+/?$")

def implausible(path: str) -> bool:
    """True if this could not be a real HBR article URL."""
    return not HBR.match(path)

implausible("/2022/04/do-boards-need-more-tech-expertise")  # False, right shape
implausible("/2022/do-boards-need-more-tech-expertise")     # True, no month

Narrow on purpose. It only works for publishers whose format is genuinely rigid. A false positive here means refusing a real citation, which is worse than the problem being solved.

When a roundup is the right tool

I am not arguing that statistics pages are worthless. They are a good way to find research and a bad way to cite it, and that distinction is the entire point.

A roundup earns its place when you use it as an index: skim it, pull the names of the underlying studies, then go read those. It is also fine when the claim you are making is about the roundup itself, or when the number is decorative rather than load-bearing. Nobody needs a primary source to say a lot of people work from home.

The line is whether the number is doing work. If your argument survives the figure being wrong by half, cite whatever you like. If it does not, you owe the reader the study.

The habit that actually helps

Never cite a number without opening the page it came from. Copy the exact digits into your notes before you write the sentence. That is the whole discipline.

It sounds trivial and it is not. The moment you try it, you discover how often the number is simply not on the page you were about to cite. The page references a study. The study is somewhere else. Sometimes it does not exist. You find this out in about forty seconds, and only if you look.

The same principle applies to anything AI-assisted, which is most research now: the tool is a way to find candidates, never a way to confirm them. Ask an agent to check a claim and it will consult the same corpus that produced the claim.

"Search does not measure whether a number is true. It measures whether the number is repeated."

Cisco Caceres•Coding since the late 1970s. Still learning, still building.