Methodology

• One prompt per model. A prompt tuned for a given model would improve its results; we measure what a standard integration gives.
• No fine-tuning, no specialised OCR upstream. A dedicated processing chain would do better.
• One hundred documents, not ten thousand. Enough to see a twenty-point gap, not to separate two models one point apart. Each score's margin of error says so.
• American invoices, in English, from a single sector: television advertising buys. Nothing guarantees these results carry over to a French invoice.
• Images, not PDFs with a text layer. A native PDF is easier for a model to read: published scores are a floor, not a ceiling.
• Two scored fields. The other fields on these invoices admit several defensible answers — six competing identifiers, two distinct periods — and the question is no longer asked.
• No line items. That is the real difficulty of the job, and the original annotation does not cover it: we would have to build our own reference.
• A task easier than expected. Twenty-five models out of twenty-seven exceed 97%: this test separates them poorly, and should be read as an entry threshold rather than a ranking.

Fields are weighted by what a mistake costs in accounting, not by how technically hard they are. One critical field wrong drops the whole invoice into “needs review”.

A field is worth its weight or zero, with no half points. Accuracy is the sum of points earned over the total possible. An invoice counts as “no review needed” when none of its critical fields is wrong, missing or invented; an invoice the model failed to process does not count.

The comparison is fussy where the business is fussy: amounts are compared to the cent. It is lenient where the business is lenient too: an identifier may be written with or without spaces, and the order of line items does not matter.

Dates follow the convention of the document's country, declared field by field in the rubric. On these American invoices, 03/04/2020 is read as 4 March; on a French invoice the same writing would mean 3 April. Reading an American date the French way had in fact produced, on a first attempt, some twenty errors wrongly blamed on the models.

A field absent from the document is part of the test. Answering “nothing” when there is nothing counts as a right answer; producing a value counts as a hallucination, tallied separately from the rest and never melted into an overall score.

• Correct: the value on the document, or “nothing” when the document contains nothing.
• Wrong or missing: a value other than the one on the document, or nothing when the information was there.
• Hallucinated: a value where the document contains none.

Only a correct field earns points. A hallucination also feeds a measure of its own: the share of genuinely absent fields that the model filled in anyway. A model that writes “non trouvé” or “n/a” is abstaining correctly: its answer is reduced to “nothing” before it is scored.

The code does not have the last word. A human arbitration stage selects the answers to review — every hallucination, those judged correct but written differently from the reference, and a sample of the rest — and the human's verdict replaces the code's in the computation.

On the published run, that stage has not been played yet: the figures shown are the comparator's alone. 80 of the 2700 gradings are flagged “needs review”, and the ranking will say otherwise once they have been arbitrated. Saying so, rather than implying a review that did not happen, is part of the protocol.

Sent as is to every model, along with the scanned pages of the invoice. The same for all twenty-seven, without a word of difference.

• Sixty-four questions. Enough to see a ten-point gap, not to separate two models one or two points apart: one question is worth 1.6 points. Each score's margin of error says so.
• The top of the ranking is not separated. Seven of twenty-seven models score 100%, and their order only reflects price then speed, which break the ties. They are the same seven on the forty-eight questions of the first draw and on all sixty-four: the sixteen added calculations did not separate them. This test tells which models fail, not which of those seven reasons best on a hard question.
• Calculations are over-represented: thirty-two questions out of sixty-four. Sixteen were added after the first run, because compound calculations are where models go wrong. All were taken, with no choice made on results, except the dividend payout ratio, dropped before the run. The first draw alone gives the same ranking (rank correlation 0.99).
• One question trips up more than half of the panel: Netflix's (EBITDA margin), missed by seventeen of twenty-seven models, one of which answered nothing. Seven add the two content-amortization lines ($3.4 billion for streaming, $79 million for DVD) to “D&A” and get 56.8% instead of 5.4%. The reference takes the single line named “Depreciation and amortization of property, equipment and intangibles”, which is what the question asks for (“D&A from cash flow statement”), and we keep it. Without that question, ten models are perfect instead of seven, and the order changes little (rank correlation 0.97).
• The useful pages are supplied. The model receives the page or pages holding the answer, not the whole report: we measure what an analyst does once the right page is open — read off, compute, judge — not the search for the information in a report of several hundred pages.
• Famous companies, reports published years ago, a public dataset since 2023: a model may know some of these figures by heart.
• American reports, in English. The question is asked in English, word for word, on both versions of the site.
• What an automatic answer can check, and nothing else: 91 questions out of 150. The other 59 — written explanations, opinions, numbers whose unit the question does not fix — are listed line by line in the published sorting, with their reason. These are the most checkable questions, not a representative sample of the job: writing a note, comparing companies, forecasting or defending a view are not graded here.
• One answer per question, written by an analyst, with whatever errors it may carry. On the 3M question about fixed assets, the reference says 8.70 billion while the page shows 8,738 million: twenty-four models answer 8.738. The 0.5% tolerance absorbs it; at 0.1%, no model is perfect any more.
• A yes or no has a fifty-fifty chance of being right by luck: the twelve verdicts weigh less than their number suggests.
• Only three questions have “there is none” as their right answer. The hallucination count is a count out of three, not a reliable percentage.

Each question is worth one point, on a single criterion: the one the question asks. A criterion that a question's ground truth does not declare is not scored for it. Without that rule it would read as “absent from the document” and earn points for a model that answered nothing.

Four criteria, one per form of answer. Reading a figure and computing a metric are compared the same way; they are kept apart so that the site can say where a model goes wrong.

A number is right within 0.5% of the reference, or within half of its last displayed digit when that is wider: the reference “1.9” accepts 1.85 to 1.95, the reference “1616” accepts 1,607.9 to 1,624.1. References are rounded by their author, and the page often carries a more precise figure: without this tolerance, a model that reads correctly would be marked wrong. It was set before the run.

The decimal point is read as such, and the “%” and “$” signs are ignored: “1.734” is 1.734, not 1,734.

A verdict is “yes”, “no” or “not_applicable”. A label is compared with a list of accepted forms, after stripping accents, case and punctuation. Equality is whole: “Corporate & Investment Bank” does not pass for “Corporate”.

The accepted forms were extended on 7 October, after the run, and that is a correction on our side. For the three questions “which activity brought in the most cash”, we expected “operations” or “operating activities”; twelve answers naming the line printed on the page — “Net cash provided by operating activities” — had been refused, among them Claude Fable 5.1's and Opus 5.5's. We added the printed labels and redid the scoring. A list of accepted answers written before seeing the answers can be too narrow; saying so is part of the protocol.

Three verdicts are used here: right, wrong or missing, and hallucinated. A question whose right answer is “there is none” is a hallucination trap: a model that names a security or a line item anyway is making it up.

Only three questions are of that kind: the listed debt securities of American Express and Ulta Beauty, and Ulta Beauty's acquisitions. Three hallucinations in all: Nova Pro makes up two, Nova Lite one. They feed their own measure and are never folded into the overall score.

A failed call does not count against the model when it comes from us: a host's rate-limit refusal, exhausted credit, an answer extractor that was too narrow. It counts against the model when it returned nothing usable: two contradictory answers, a subtraction instead of a number, thinking that never ends. Five cases, each read and explained on the benchmark page: Mistral Large three times, Kimi K3 once, Ministral 8B once. The model is marked “missing” there, and it alone.

This rule was adopted after the first run. The previous one removed the question for the whole panel as soon as one model did not answer it, which made the most disputed questions drop out, Netflix's among them: the test became easier. In two cases Mistral Large's last answer was right; we do not decide for it.

The code does not have the last word, but human arbitration was not played on this run: the figures shown are the comparator's alone. The three hallucinations and a control sample are waiting for it. As written, the step does not present answers judged wrong: a label the comparator does not recognise does not go through it, which is how a too-narrow list of forms could go unnoticed until we looked at the answers question by question.

The template sent to every model, with the question and the report pages as images. Only the format instruction for the relevant form — a number, a verdict or a label — goes out with the question. The same for all twenty-seven, not a word different.

One sentence of the instruction had an effect we had not foreseen: “a percentage of 12.5% is written 12.5” pushed 21 of 23 models to write Coca-Cola's return on assets as a percentage, when the reference was the rounded ratio, 0.01. That question, its twin on AES and General Mills' retention ratio were dropped after the runs: they measured the scale chosen, not the computation. The prompt was not changed, so that every published figure comes from the same text.

Sixty-seven questions prepared, sixty-four ranked. The first run, on 7 October, asked fifty questions drawn by quotas; the second, on 8 October, seventeen added calculations. Three were dropped afterwards for their ambiguous scale: the two returns on assets and the retention ratio. All are listed, with their reason, on the benchmark page, together with the five failures counted against their model.

Some calls were replayed: right answers our extractor had failed to read, rate-limit refusals at a host, credit exhausted mid-run, a raised token ceiling for four models that think at length. The two runs cost about $12.6 charged, against $12.1 tracked by the pipeline: the gap comes from interrupted calls the provider bills without our recording them.