THINK AI FIRST

Dataset

Invoice extraction by three low cost LLMs

288 raw model responses, 24 script generated invoices with their ground truth, four levels of degradation. Test conducted on 2026-08-10.

This page describes one dataset, published in full. It is the raw material behind our measurement of silent extraction errors: 24 supplier invoices, four levels of text degradation, three low cost models, and the 288 responses those models actually returned.

Everything on this page comes from that one run. Nothing is estimated.

What it measures, and what it does not prove

It measures one thing precisely: how often an extraction is wrong, silent, and plausible. Wrong, meaning at least one of seven fields does not match the ground truth. Silent, meaning the model filled no warning in the notes field it was explicitly offered. Plausible, meaning subtotal + tax = total_due still holds, so the cheapest available check does not fire.

What it does not prove, stated plainly, because a dataset quoted without its limits does more harm than no dataset at all:

  • The input is text, not scanned images. It says nothing about OCR character errors, which can only make results worse.
  • The corpus is built by script, not collected. No real invoice, no real supplier.
  • 24 invoices. A test on 24 cases is a test on 24 cases.
  • One run per combination, at temperature 0. Gaps of one or two points between models are not significant here. The shape of the errors is what holds.
  • English only, one page per invoice.
  • Three entry level models. Nothing here says what more expensive ones would do.

The corpus is built, and here is what that means

The 24 invoices were generated by script with the fixed seed 20260810. That is a deliberate trade, and it cuts both ways.

What it buys: the ground truth is known without argument, and anyone can regenerate the exact same corpus and disagree with our reading of it. What it costs: these are not real documents, and the layouts a real supplier produces are more varied than anything a generator invents.

Four difficulties were inserted, each of them resolved by the document itself so that a single defensible answer exists: a deposit already paid and deducted, a discount applied before tax, an ambiguous currency symbol resolved in words, and an ambiguous date format resolved on the next line.

Test conducted on 2026-08-10. This page was published on 2026-08-13. The extractions were run by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, and all of them are in the archive below.

Raw figures

Invoices fully correct, out of 24

Model N1 clean N2 columns lost N3 totals shuffled N4 labels lost
openai/gpt-4o-mini 24/24 24/24 21/24 14/24
google/gemini-2.5-flash-lite 24/24 24/24 18/24 19/24
anthropic/claude-haiku-4.5 24/24 24/24 12/24 18/24
All models 72/72 72/72 51/72 51/72

Silent failure

Level Wrong extractions Wrong with no warning Share
N1 clean 0 0 not applicable
N2 columns lost 0 0 not applicable
N3 totals shuffled 21 3 14.3 percent
N4 labels lost 21 10 47.6 percent

The arithmetic check, subtotal + tax = total due

Level Wrong ones caught Catch rate Correct ones flagged wrongly False alarm rate
N1 clean 0 not applicable 18 25.0 percent
N2 columns lost 0 not applicable 18 25.0 percent
N3 totals shuffled 10 47.6 percent 6 11.8 percent
N4 labels lost 15 71.4 percent 12 23.5 percent

Which field breaks

Field N1 N2 N3 N4
invoice_number, invoice_date, due_date, currency 0 0 0 0
subtotal 0 0 0 17
tax 0 0 0 3
total_due 0 0 21 10

Run totals

Quantity Value
Extractions 288
Technical failures 0
Invalid JSON responses 0
Total API cost 0.11951 USD
Wrong, silent and passing the arithmetic check 8

What the archive contains

Seven files, 393777 bytes uncompressed. The inventory is here so you know what you are downloading before you download it.

File What it holds Size
generer-corpus.py the corpus generator, fixed seed 20260810 7.2 KiB
corpus.json the 24 invoices and their ground truth 28.6 KiB
degrader.py the four degradation levels 2.6 KiB
conduire-test.py the model calls and the field by field comparison 6.6 KiB
resultats-bruts.json the 288 responses, raw, with the prompt used 326.7 KiB
analyser.py the computation of every figure on this page 4.6 KiB
chiffres.json the computed figures 8.4 KiB

The scripts and their comments are in French, which is the working language behind this site. The API key is read from an environment variable and appears in none of these files.

Rerun it

The generator uses a fixed seed, so the corpus you produce is the corpus we used.

bash
unzip 2026-08-invoice-extraction.zip
cd 2026-08-invoice-extraction
python3 generer-corpus.py --sortie corpus.json
python3 conduire-test.py --corpus corpus.json --sortie resultats-bruts.json \
    --niveaux 1 2 3 4 \
    --modeles openai/gpt-4o-mini google/gemini-2.5-flash-lite anthropic/claude-haiku-4.5
python3 analyser.py --resultats resultats-bruts.json --sortie chiffres.json

Each raw response is one line of resultats-bruts.json, with the fields the model returned, which of them matched, whether it warned about anything, and whether the arithmetic check passed:

json
{"modele": "openai/gpt-4o-mini", "facture": "F04", "niveau": 3,
 "json_valide": true, "nb_champs_justes": 6, "tous_les_champs_justes": false,
 "a_signale_quelque_chose": false, "controle_arithmetique_passe": true}

Verify what you downloaded before you trust it:

bash
shasum -a 256 2026-08-invoice-extraction.zip
# 8d8982b037ddb6b0a1c6c420375024e6d63bfc9aff8852c49c8a22766ecddd84

How to cite this dataset

Copy one of the two forms below. The reference points at this page, which is the stable address of this dataset; the archive is what that address serves.

citation
Think AI First (2026). Invoice extraction by three low cost LLMs, 288 raw responses.
Think AI First. Test conducted 2026-08-10, published 2026-08-13.
https://think-ai-first.com/datasets/invoice-extraction-2026-08. Licensed CC BY 4.0.
bibtex
@dataset{thinkaifirst2026invoice,
  author    = {{Think AI First}},
  title     = {Invoice extraction by three low cost LLMs, 288 raw responses},
  year      = {2026},
  publisher = {Think AI First},
  url       = {https://think-ai-first.com/datasets/invoice-extraction-2026-08},
  note      = {Test conducted 2026-08-10. Licensed CC BY 4.0. Archive SHA-256 8d8982b037ddb6b0a1c6c420375024e6d63bfc9aff8852c49c8a22766ecddd84}
}

Licence and reuse

Published under CC BY 4.0. Use it, quote it, contradict it. If you rerun this and get something different, tell us through the contact form: corrections are published on the article itself, dated.

Download the dataset (zip, 27.8 KiB, 7 files)

Other datasets published here

Each one is a separate test, with its own corpus, its own raw responses and its own archive.