Price against accuracy on invoice extraction, six models

This page describes one dataset, published in full. It is the raw material behind our measurement of price against accuracy on invoice extraction: 864 calls, six models paired by vendor and generation, the cost actually billed for every call, and the two passes it took to get an honest comparison.

Everything on this page comes from that one run. Nothing is estimated.

What it measures, and what it does not prove

It measures one thing: what a tier upgrade buys, and what it costs, on a task three low cost models already do reasonably well. Six models, three vendors, three pairs. Inside a pair the vendor and the generation are held constant and only the tier changes, so the difference cannot be read as a difference of generation.

The price figure is not taken from a price list. It is the amount actually billed for the call, read out of the API response, reasoning tokens included. That distinction is the whole point of the test: one of the three premium models costs ten times what its advertised ratio suggests.

What it does not prove, stated plainly, because a dataset quoted without its limits does more harm than no dataset at all:

• 24 script generated invoices are not a sample of real invoices. No scan, no foreign language, no exotic layout, no handwritten stamp.
• One call per case. The accuracy rates carry a sampling uncertainty that is not measured here, and our own test of 2026-08-25 established that temperature 0 does not guarantee repeatability.
• API prices change. The costs are those of 2026-09-06 and nothing else.
• A perfect score is 96 out of 96, not a guarantee. One model got every invoice right on this corpus. That is a fact about 96 cases.
• Nothing here says a premium model stays better on another task.

The corpus is reused, and here is what that means

The 24 invoices, their four degradation levels and their ground truth are exactly those published here on 2026-08-13, under the invoice extraction dataset, generated by script with the fixed seed 20260810. They were not regenerated, and the instruction sent to the models is the August one word for word, at temperature 0.

That choice is deliberate and it is why the comparison means anything: all six models judge an identical set of documents. Regenerating would have mixed the effect of the tier with the effect of a different corpus, and no figure would separate the two.

It also lets this dataset answer a question the previous ones left open. Pass A replays the August protocol to the token, 27 days later, on the three economy models: same verdict on 287 of 288 cases, identical values on 282. The August dataset is still a fair description of what those models do.

Two passes, and a failed one kept as it is

Pass A, output budget 700 tokens, three economy models, 288 calls. Drift witness.

Pass B, output budget 4000 tokens, six models, 576 calls. The accuracy comparison is read here. A 700 token budget truncates a reasoning model, which would have produced a false accuracy result where there was only a budget that was too short.

On the first run of pass B, 81 of the 96 claude-opus-4.5 calls failed with HTTP 402: the remaining credit did not cover the calls in flight. The harness counts a technical failure as a wrong extraction, so the model appeared to be the worst of the six. Those 96 calls were replayed alone, and the model returns 89 of 96.

Both files are in the archive. passeB-six-modeles-budget4000.json holds the failed run exactly as it happened, passeB-opus45-budget4000.json the replay, and passeB-fusionnee.json the merged valid pass with its merge note. A test that hides its failed measurement is worth less than one that shows it.

Raw figures

Accuracy is the share of invoices whose seven fields are all correct, out of 96 cases per model. Cost is dollars actually billed, scaled to 1000 invoices.

What the upgrade buys, pair by pair.

Accuracy per model and per degradation level, 24 invoices each.

The model's own spontaneous note, scored on both of its rates.

Drift at 27 days, pass A against the raw August results on the same 288 calls.

What the archive contains

Ten files, 2.0 MB uncompressed. In reading order: what builds the corpus, what calls the models, what those models actually returned, and what analyses it. The corpus is shipped with the archive rather than referenced, so this archive stands on its own without fetching another one.

The scripts and their comments are in French, which is the working language behind this site. The API key is read from an environment variable and appears in none of these files.

Rerun it

The analysis calls no model at all, so it replays at zero cost on the raw responses shipped in the archive. Only conduire-test.py spends anything.

unzip 2026-09-invoice-price-accuracy.zip
python3 analyser.py --passeB passeB-fusionnee.json \
    --passeA passeA-economie-budget700.json --sortie chiffres.json

Each raw response is one entry of the pass files, exactly as the model returned it, with the cost the API billed for it. This one is the reasoning model on a clean invoice, and it is the shape of the false alarm counted in the table above: all seven fields correct, 1206 reasoning tokens paid for, and a note that reports two facts about the document without claiming any field is wrong. Scored as a flag, it costs a human a re-read for nothing.

{
 "modele": "google/gemini-2.5-pro",
 "facture": "F01",
 "niveau": 1,
 "lu": true,
 "json_valide": true,
 "tous_les_champs_justes": true,
 "jetons_raisonnement": 1206,
 "cout_usd": 0.014135,
 "note_du_modele": "The first two line items have the same description but a significant difference in unit price ($5.57 vs $144.51). The currency '$' was assumed to be USD."
}

Verify what you downloaded before you trust it:

shasum -a 256 2026-09-invoice-price-accuracy.zip
# 80bb17e8c06d57cb4ba3e6152a9fb4bb417ec298b3ce346d2f614f81f872acbb

How to cite this dataset

Copy one of the two forms below. The reference points at this page, which is the stable address of this dataset; the archive is what that address serves.

Think AI First (2026). Price against accuracy on invoice extraction, six models, 864 raw responses with their billed cost.
Think AI First. Test conducted 2026-09-06, published 2026-09-07.
https://think-ai-first.com/datasets/invoice-price-accuracy-2026-09. Licensed CC BY 4.0.

@dataset{thinkaifirst2026priceaccuracy,
  author    = {{Think AI First}},
  title     = {Price against accuracy on invoice extraction, six models, 864 raw responses with their billed cost},
  year      = {2026},
  publisher = {Think AI First},
  url       = {https://think-ai-first.com/datasets/invoice-price-accuracy-2026-09},
  note      = {Test conducted 2026-09-06. Licensed CC BY 4.0. Archive SHA-256 80bb17e8c06d57cb4ba3e6152a9fb4bb417ec298b3ce346d2f614f81f872acbb}
}

Licence and reuse

Published under CC BY 4.0. Use it, quote it, contradict it. If you rerun this and get something different, tell us through the contact form: corrections are published on the article itself, dated.

Download the dataset (zip, 109.8 KiB, 10 files)

Other datasets published here

Each one is a separate test, with its own corpus, its own raw responses and its own archive. This one reuses the corpus of the first and prices what a tier upgrade buys on it.

• Invoice extraction by three low cost LLMs, 288 raw responses, test conducted 2026-08-10.
• Contact list deduplication by three low cost LLMs, 96 raw responses, test conducted 2026-08-17.
• Meeting action item extraction by three low cost LLMs, 96 raw responses, test conducted 2026-08-18.
• Repeatability of an LLM step at temperature 0, 360 raw responses, test conducted 2026-08-25.
• Four error detection guardrails on 288 invoice extractions, 288 raw self-check responses, test conducted 2026-09-04.
• Two error detection guardrails on 368 proposed contact merges, 96 raw self-check responses, test conducted 2026-09-12.
• Invoice against purchase order matching by four LLMs, 672 raw decisions, test conducted 2026-09-14.