Runnable protocolOn documents

Advertising invoice extraction

How many invoices does a model read with nothing for a human to fix?

Updated 5 October 2026 · Run 2026-09-27_facture-fcc+2026-10-02_facture-fcc+2026-10-04_facture-fcc+2026-10-05_facture-fcc · 27 models · 100 invoices

Advertising invoice extractionAccuracy × cost per test

Each square is a model, in its lab's colour and monogram. Click to open its page.

Key takeaways

5 models tie for first place at 100.0% accuracy: GPT-6 Luna, DeepSeek V4.1 Flash, Command A+, Kimi K2.6 and Muse Spark 1.3. This test does not separate them; the table orders them by price, then by speed.
Best accuracy for the price: Gemma 4 31B, 0.7 points behind the best score at $0.0003 per invoice — 3.4 times cheaper than GPT-6 Luna.
24 of 27 models invent nothing: 0% hallucinations.
The hardest field: “Total amount billed”, answered correctly 97.0% of the time on average across the models.

Which model to choose

If accuracy comes first

GPT-6 Luna

100.0% accuracy

If volume matters

Gemma 4 31B

$0.0003 per invoice

If errors are costly

GPT-6 Luna

0.0% invented items

The leaderboard

Click a header to re-sort. Accuracy orders the table; the other measures are never blended into it.

Advertising invoice extractionAccounting

Task

Rank	Model (27)					Input / output price
1	GPT-6 Luna	100.0%±0.0	100.0%	0.0%	$0.0009	$0.1 / $0.5	4.0 s
2	DeepSeek V4.1 Flash	100.0%±0.0	100.0%	0.0%	$0.0025	$0.3 / $1.2	12.0 s
3	Command A+	100.0%±0.0	100.0%	0.0%	$0.0060	$0.3 / $1.5	14.4 s
4	Kimi K2.6	100.0%±0.0	100.0%	0.0%	$0.013	$0.434 / $1.828	28.3 s
5	Muse Spark 1.3	100.0%±0.0	100.0%	0.0%	$0.013	$1.25 / $4.25	10.4 s
6	Kimi K3	99.8%±0.7	100.0%	1.0%	$0.021	$2.7 / $13.5	7.0 s
7	Gemma 4 31B	99.3%±1.2	99.0%	0.0%	$0.0003	$0.09 / $0.34	4.4 s
8	Qwen3.7 Flash	99.3%±1.2	99.0%	0.0%	$0.0005	$0.03 / $0.13	36.2 s
9	Ministral 3 8B 2512	99.3%±1.2	99.0%	0.0%	$0.0010	$0.15 / $0.15	2.3 s
10	GLM 5.3 Flash	99.3%±1.2	99.0%	0.0%	$0.0012	$0.15 / $0.5	6.6 s
11	Gemini 3.5 Flash Lite	99.3%±1.2	99.0%	0.0%	$0.0015	$0.3 / $2.5	1.7 s
12	Qwen3.8 27B	99.3%±1.2	99.0%	0.0%	$0.0036	$0.42 / $3	16.1 s
13	Gemini 3.8 Flash	99.3%±1.2	99.0%	0.0%	$0.0070	$0.75 / $3.75	6.5 s
14	Claude Sonnet 5	99.3%±1.2	99.0%	0.0%	$0.015	$2 / $10	2.9 s
15	Qwen3.8 Max (0902)	99.3%±1.2	99.0%	0.0%	$0.020	$2 / $6	14.8 s

What this benchmark measures

Real television advertising invoices filed with the US telecom regulator and annotated by hand by journalists. The model receives the scanned pages and must return two things: the total amount billed, and the VAT number — the latter absent from these documents, to see who invents rather than abstains. Every answer is compared against the original annotation by a hand-written comparator: no AI grades an AI.

Where the documents come from

Invoices filed by US television stations with the Federal Communications Commission, which law makes public. The annotations come from journalists who entered them by hand to track campaign spending.

Source: DeepForm — archives publiques de la FCC
Licence: MIT (outillage) · documents publics déposés auprès de la FCC

Sample: 100 invoices
What the model receives: Documents (images or PDF)
Models tested: 27
Protocol: Runnable protocol
Status of the figures: Real measurement
Run: 2026-09-27_facture-fcc+2026-10-02_facture-fcc+2026-10-04_facture-fcc+2026-10-05_facture-fcc

Where models pull apart

The average across all models on each graded field, from best mastered to hardest, and the best model on each.

VAT number (absent from the document)99.9%GPT-6 Luna100.0%
Total amount billed97.0%GPT-6 Luna100.0%

The documents that split the models

These invoices are real: no difficulty was planted in them. They are the ones the models contradicted each other on most. Here is the page, the original annotation, and what each model answered — enough to judge the scoring without taking our word for it.

Montant total facturé

13 of 27 models got it wrong · field: Montant total facturé

Test invoice 9ed3362a-84b4-1d8e-7aab-1ed898008b16 — 6 scanned pages · 9ed3362a-84b4-1d8e-7aab-1ed898008b16

Original annotation: 6414
Nova Lite 1.0: 6414
Nova Pro 1.0: 5451.9
Claude Fable 5.1: 9347
Claude Opus 5.5: 9347
Claude Sonnet 5: 6414
Command A+: 6414
DeepSeek V4.1 Flash: 6414
Gemini 3.5 Flash Lite: 6414
Gemini 3.8 Flash: 9347
Gemma 4 31B: 6414
Llama 4 Maverick: 6414
Muse Spark 1.3: 6414
Ministral 3 8B 2512: 6414
Mistral Large 3 2512: 5451.9
Mistral Medium 3.5: 6414
Mistral Small 4: 6414
Kimi K2.6: 6414
Kimi K3: 6414
GPT-6 Astra: 9347
GPT-6 Luna: 6414
GPT-6 Sol: 9347
Qwen3.7 Flash: 9347
Qwen3.8 27B: 9347
Qwen3.8 Max (0902): 9347
Qwen3.8 Max Prime: 9347
Grok 4.7: nothing — the model abstains
GLM 5.3 Flash: 9347

Montant total facturé

9 of 27 models got it wrong · field: Montant total facturé

Test invoice 93cd08d3-28b1-31c0-8902-4060655fcb91 — 2 scanned pages · 93cd08d3-28b1-31c0-8902-4060655fcb91

Original annotation: 4720
Nova Lite 1.0: 4838
Nova Pro 1.0: 4012
Claude Fable 5.1: 4838
Claude Opus 5.5: 4720
Claude Sonnet 5: 4838
Command A+: 4720
DeepSeek V4.1 Flash: 4720
Gemini 3.5 Flash Lite: 4838
Gemini 3.8 Flash: 4720
Gemma 4 31B: 4720
Llama 4 Maverick: 4838
Muse Spark 1.3: 4720
Ministral 3 8B 2512: 4720
Mistral Large 3 2512: 4720
Mistral Medium 3.5: 4838
Mistral Small 4: 4838
Kimi K2.6: 4720
Kimi K3: 4720
GPT-6 Astra: 4720
GPT-6 Luna: 4720
GPT-6 Sol: 4838
Qwen3.7 Flash: 4720
Qwen3.8 27B: 4720
Qwen3.8 Max (0902): 4720
Qwen3.8 Max Prime: 4720
Grok 4.7: 4720
GLM 5.3 Flash: 4720

Montant total facturé

6 of 27 models got it wrong · field: Montant total facturé

Test invoice 4b703aaa-2e13-528c-85f8-9bd7879d4cbb — 4 scanned pages · 4b703aaa-2e13-528c-85f8-9bd7879d4cbb

Original annotation: 7000
Nova Lite 1.0: 6500
Nova Pro 1.0: 5950
Claude Fable 5.1: 7000
Claude Opus 5.5: 7000
Claude Sonnet 5: 7000
Command A+: 7000
DeepSeek V4.1 Flash: 7000
Gemini 3.5 Flash Lite: 7000
Gemini 3.8 Flash: 7000
Gemma 4 31B: 7000
Llama 4 Maverick: 6500
Muse Spark 1.3: 7000
Ministral 3 8B 2512: 6500
Mistral Large 3 2512: 6500
Mistral Medium 3.5: 7000
Mistral Small 4: 6500
Kimi K2.6: 7000
Kimi K3: 7000
GPT-6 Astra: 7000
GPT-6 Luna: 7000
GPT-6 Sol: 7000
Qwen3.7 Flash: 7000
Qwen3.8 27B: 7000
Qwen3.8 Max (0902): 7000
Qwen3.8 Max Prime: 7000
Grok 4.7: 7000
GLM 5.3 Flash: 7000

The documents analysed

The 100 documents in the test, from the most-missed to the most agreed-on: the annotation used as reference, how many models found it, and a link to the original where it is published. The ranking does not ask to be believed: it can be recounted.

The documents analysed
Document	Expected Total amount billed	Models correct	Invented value	Original
9ed3362a	6414	14 / 27	—	open
93cd08d3	4720	18 / 27	—	open
4b703aaa	7000	21 / 27	—	open
15262983	17650	23 / 27	—	open
17bee528	2095	24 / 27	—	open
3fb2b31f	209250	24 / 27	—	open
2388be11	995	25 / 27	—	open
4809552b	350	25 / 27	—	open
4e1c4012	2255	25 / 27	—	open
5d56f8db	12100	25 / 27	—	open
64780ed0	40700	25 / 27	—	open
7bbe366d	4485	25 / 27	—	open
afa6fb20	21300	25 / 27	—	open
d2bbcfb2	355	25 / 27	—	open
dd623c62	40700	25 / 27	—	open
03b940c2	16350	26 / 27	—	open
08bea544	27950	26 / 27	—	open
0be55a7b	1428	26 / 27	—	open
0e4719a3	6680	26 / 27	—	open
250e73dd	3755	26 / 27	—	open
26f3961b	34900	26 / 27	—	open
3f335f42	6545	26 / 27	—	open
4cee59d1	3685	26 / 27	—	open
59d47fbe	9580	26 / 27	—	open
5b40f242	5360	26 / 27	—	open
5d8fb479	3526	26 / 27	—	open
703ebcab	212	26 / 27	—	open
74c779b4	72500	26 / 27	—	open
97ea6a67	7600	26 / 27	—	open
98905496	1880	26 / 27	—	open
991a6b5e	1720	26 / 27	—	open
9cc36a83	440	26 / 27	—	open
a5b12494	6050	26 / 27	—	open
bc343c38	625	26 / 27	—	open
c6ec7473	3625	26 / 27	—	open
d6bd95e8	2425	26 / 27	—	open
d748c964	194	26 / 27	—	open
e3f3dba6	60	26 / 27	—	open
f9889d44	1900	26 / 27	—	open
02ccb7f2	210350	27 / 27	—	open
0458e5e4	2120	27 / 27	—	open
05a53fa5	6025	27 / 27	—	open
0617c152	7465	27 / 27	—	open
0880bc9a	44730	27 / 27	—	open
0a32ce11	1425	27 / 27	—	open
11a46278	8260	27 / 27	—	open
1fd3058f	11490	27 / 27	—	open
1fd7b5c9	43215	27 / 27	—	open
21228dd6	3060	27 / 27	—	open
29604390	42750	27 / 27	—	open
2aac88c6	5780	27 / 27	—	open
331ff2f3	22300	27 / 27	—	open
345c9cc0	15500	27 / 27	—	open
3499dd36	8670	27 / 27	—	open
36ff284e	2250	27 / 27	—	open
3bdbc1a7	3384	27 / 27	—	open
48e3a952	14685	27 / 27	—	open
5c8edc38	3110	27 / 27	—	open
5e0773ca	34500	27 / 27	—	open
627d67b5	22180	27 / 27	—	open
62e6d540	4850	27 / 27	—	open
6c10638e	2200	27 / 27	—	open
707c944c	29175	27 / 27	—	open
73e15945	28700	27 / 27	—	open
76b092d7	3035	27 / 27	—	open
7b9146bb	1025	27 / 27	—	open
7e00d875	360	27 / 27	—	open
805aafc6	16045	27 / 27	—	open
8222d0e9	10095	27 / 27	—	open
8588921b	3120	27 / 27	—	open
88d79dde	865	27 / 27	2	open
8b687291	8850	27 / 27	—	open
8e41b954	179980	27 / 27	—	open
9215778a	3500	27 / 27	—	open
9294d8b3	15070	27 / 27	—	open
988878c1	3675	27 / 27	—	open
a21fa434	39500	27 / 27	—	open
a24ef1d8	21960	27 / 27	—	open
a7d60614	4820	27 / 27	—	open
a8aaf9d5	14150	27 / 27	—	open
abf2f735	1690	27 / 27	—	open
b34fc957	2635	27 / 27	—	open
be9e50f7	6200	27 / 27	—	open
c0b67c9f	50277	27 / 27	—	open
c6974e8e	235350	27 / 27	—	open
ca356a4b	479	27 / 27	—	open
d06fd8db	1800	27 / 27	—	open
d1ba9225	24380	27 / 27	—	open
d297f399	6925	27 / 27	—	open
d6cd4fa8	800	27 / 27	—	open
da77aaa0	55600	27 / 27	—	open
db47993a	5470	27 / 27	—	open
e0260276	4455	27 / 27	—	open
e5e2f343	32200	27 / 27	—	open
e6846bc9	4215	27 / 27	—	open
e8d41204	73200	27 / 27	—	open
ec8415e2	1213	27 / 27	—	open
ee19ec76	80	27 / 27	—	open
f56e2fc1	2690	27 / 27	1	open
f5871a46	15150	27 / 27	—	open

How these figures are produced — the scoring grid, the exact prompt, and what this test does not measure.