Analysis
Human review: the six “inventions” were printed on the page
The comparator had counted six hallucinations across our two benchmarks. A human opened the six documents: every value is there; it simply does not answer the question. The six answers remain wrong, and no model is counted as inventing any more.
Our two rankings showed a non-zero hallucination rate for five models. On the invoices, Kimi K3, Llama 4 Maverick and Mistral Medium 3.5 gave a VAT number for American invoices that carry none. On the financial reports, Nova Pro and Nova Lite named a security where the analyst answers “there is none”.
Those verdicts came from the comparator, and the protocol provides for a human to review them. That has been done since 9 October, with the documents open.
What the pages carry:
- Invoices: the “VAT number” given by the three models is printed on the invoice. It is the TV station's federal tax identifier, labelled “FIIY #” on one and “TIN” on the other.
- American Express: asked “which debt securities are listed?”, Nova Pro and Nova Lite answer “Common Shares”, the only line in the table of listed securities. Those are shares, not debt.
- Ulta Beauty: asked the same question, Nova Pro answers “The Nasdaq Global Select Market”, the name of the exchange, printed on the same line.
None of these six values is invented. All are wrong: the model took from the page something other than what it was asked for. We now count them that way. Accuracy scores do not move, since a wrong answer and an invented one are both worth zero. The hallucination rate, though, drops to zero for all twenty-seven models on both tests.
Zero does not mean these models never invent. The finance test has only three questions whose right answer is “there is none”, and the invoices a single absent field.
What we had written that was wrong: that three models “invent a VAT number that does not exist”, and that two others “fabricate” an answer. Both articles are corrected, and the correction is dated in each.
The lesson on method: a comparator knows that a value was given where the reference says “nothing”. It does not know whether that value is on the page. A “hallucinated” verdict coming out of the code is therefore a presumption, which only a human opening the document can confirm.
The review also confirmed the answers accepted despite a difference in spelling from the reference: “5.43” for “5.4”, “Net cash provided by operating activities” for “operations”. It did not cover the random check of answers identical to the reference, which remains to be done.