Our first accuracy measurement returned 99.5%. It was false
A metric that cannot come out wrong measures nothing. It measures that whoever wrote it chose the right words.
Of the 107 checks producing that 99.5%, 106 could not fail by construction. Once every claim had to be contradictable by the answer, the real figure came out at 64.2% over 53 checks.
- Sample
- 107 initial checks · 53 after requiring every claim to be falsifiable
- Period
- August – September 2026
- Method
- Every AI claim about a verified brand fact scores 2 (correct), 1 (partially correct) or 0 (contradicts). Silence does not score.
- Language
- Spanish (Spain)
What was wrong
The Accuracy Score measures how often an AI gets a fact about you right when it states one. The first version scored two kinds of claim: value claims (a year, a city, a figure) and text claims (whether the answer contains a given expression). Text claims only declared what should appear, never what should not.
A claim that can only be right or not applicable can never be wrong. It can only push the percentage up.
The extreme case was tautological: one check fired when the sentence contained the word "geo" and then checked whether the sentence contained the word "geo". A guaranteed hundred percent — and 63 of the 107 checks were that one.
The fix
It was fixed in the module and not just in the data file, which was the easy temptation. A claim now only enters the score if it is falsifiable: value claims are by nature; text claims only if they also declare what expression contradicts them. Discarded claims are reported separately, with the reason and with how many times they would have scored — which is the number that convinces you to fix them.
With the filter in place, the sample dropped from 107 to 1. Fifty percent over one check is not publishable either, so a minimum of 10 observations was added, below which the metric is shown but gives no verdict.
What appeared once the number was real
With facts verified by the brand owner, Accuracy came out at 64.2% over 53 checks, with 9 real errors. And there was the finding no other view would have produced:
- 8 of the 9 errors were the same fact: years of experience.
- 7 of those answers cited the brand's own website. The site said "10+ years" and the real figure was more than fifteen. The AI was not inventing the error: we were feeding it.
Two false positives you only catch by hand
- Homonyms. A short name fits whole inside a longer one, so sentences about a different person with the same name plus one more surname were counted as engine errors. There were 36 of them: the worst possible false positive, because it punished the engine precisely when it was disambiguating correctly.
- An exclusion that was too broad. "Foundations" was declared as something the brand does not do — and it does work with foundations. The rule that came out of it: what you declare as not yours is the trade that is not yours, never the sector your clients are in. With no exclusions the percentage rises on its own; with exclusions that are too broad it flags as an error something that is true. Both extremes lie.
What this finding does NOT prove
- It is one brand and 53 checks. The 64.2% describes that case, not the general state of AI accuracy.
- It does not measure hallucinations in general, only contradictions with facts the brand owner has verified in writing.
- Silence does not score: a model not mentioning a fact about you counts neither for nor against. Counting it would turn the metric into noise.
- It does not say other tools have this flaw. It says we had it, and how it is detected.
Frequently asked questions
What is the Accuracy Score?
It is how often AI gets a specific fact about your brand right: 2 points if correct, 1 if partially correct, 0 if it contradicts a verified fact. Silence does not score, and the percentage is always published with its number of observations and its confidence interval.
Why is a 99.5% accuracy figure suspicious?
Because it almost always means the checks could not fail. Before believing an accuracy figure, ask what answer would have brought it down: if no such answer exists, the number is not measuring accuracy, it is measuring how the check was worded.
Can AI get it wrong because of my own website?
Yes, and more often than you would think. In this case, 7 of the wrong answers cited the brand’s own site as their source: the outdated fact was on their own page. Fixing the source is cheaper than trying to fix the model.