Invoice extraction by LLM: 288 tests on how often a wrong answer looks right

A small business that receives supplier invoices as PDFs would rather have a model read them than retype them. Whether that works is not really the question. On clean documents it works, and we measured that too. The question is what happens on the day it does not work, and whether anybody notices.

We ran 288 extractions to find out. Here are the results, before the method.

On clean text, and on text with the column layout destroyed, three cheap models got 144 out of 144 extractions exactly right. Losing the alignment of a PDF copy and paste cost nothing at all. The common advice to "clean up the document first" is not supported by this measurement.

When the totals block was reordered, or stripped of its labels, accuracy fell to 51 out of 72 at each of those two levels, and the models went quiet exactly where they should have spoken. At the harder level, 47.6 percent of the wrong answers came back with no warning of any kind, against 14.3 percent at the easier one. The silence grows with the difficulty.

Eight extractions out of 288 were wrong, carried no warning, and passed the arithmetic check that is supposed to catch them. All eight were invoices with a deposit already paid, and in every case the model returned the full invoiced amount instead of the balance due. A business paying on that basis would pay the deposit twice, somewhere between 252.95 and 602.26 USD per invoice, median 489.83 USD, with nothing anywhere to signal it.

Method, in short

Test conducted on 2026-08-10 by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, and they are published below so that anyone can rerun them and disagree with the result. Etienne signs this article and answers for its content.

The corpus is built, not collected. 24 supplier invoices generated by script with a fixed seed, so the ground truth is known without argument and the corpus regenerates identically. No real invoice, no real supplier, no real company data. That is a deliberate trade: a corpus of real invoices would have required us to settle 168 values by hand, with our own reading errors inside the reference.

Seven fields are extracted from each invoice: invoice number, invoice date, due date, currency, subtotal, tax, total due.

Each invoice was presented at four levels of text degradation. We degrade the structure, never the figures: corrupting an amount would make the invoice genuinely wrong and would punish a model for reading it correctly. At every level the right answer stays derivable from the document.

Three models, picked among those a small business would actually use for their price: openai/gpt-4o-mini, google/gemini-2.5-flash-lite, anthropic/claude-haiku-4.5. Access through OpenRouter, temperature 0, one run per combination of invoice, level and model. 288 calls, 0 technical failures, 0 invalid JSON responses, total API cost 0.11951 USD. The instruction was identical word for word across the three models, and it explicitly allowed the model to flag an ambiguity in a notes field. That field is what makes the silent failure rate measurable at all.

What this does not prove. The input is text, not a scanned image, so it says nothing about character level OCR errors, which can only make things worse. Every invoice is one page and in English. Each combination was run once. This is a test on 24 synthetic documents, not a production benchmark. The full list of limits is at the end, and it is worth reading before quoting any figure here.

Four traps, and the one we got wrong

Four difficulties were inserted, each of them resolved by the document itself, so that a single defensible answer exists.

The ambiguous date never fooled anybody, at any level, on any model: 0 errors out of 72 extractions. It is the trap we thought was the most dangerous when we wrote the corpus. We were wrong about it, and we are leaving that here rather than quietly dropping the row.

The invoices with no trap at all never broke either. Every single error in this test landed on an invoice where money was genuinely at stake.

Raw results

Accuracy: invoices fully correct, out of 24

The ranking inverts between N3 and N4. gpt-4o-mini is the best at N3 and the worst at N4; claude-haiku-4.5 does the opposite. Picking a model on one level of degradation tells you nothing about its behaviour on another.

Silent failure

The cheapest check: subtotal + tax = total due

The false alarms are not random noise. They are exactly the invoices with a deposit, where subtotal plus tax equals the invoiced total and not the balance due. The check is arithmetically right and operationally wrong on those, which is worth knowing before you wire it to an alert.

Which field breaks

At N3 the 21 errors all land on a single field, total_due. Nothing else moves. The identifiers and the dates survive every level of degradation we threw at them.

The blind spot

Three things have to line up for an error to reach the accounting system unchallenged: the extraction is wrong, the model says nothing, and the arithmetic check passes. That happened 8 times out of 288.

Eight extractions, five distinct invoices, all of them invoices carrying a deposit, and all eight produced by the same model, gpt-4o-mini. We report that last point because it is in the data, and we discount it in the same breath: with a single run per combination, a model level claim on eight cases is an observation, not a result. What is a result is the shape. Every case is the same mistake, on the same kind of document, on the one field where the arithmetic check cannot help, because the returned figure is internally consistent. It just answers a different question than the one asked.

The usable rule

Two things come out of this, and both are cheap.

Run the subtotal + tax = total due check. It is three lines of code and it catches half to two thirds of the errors. Accept that it will flag around a quarter of your correct invoices at the clean levels, and route those to a human rather than trying to tune it.

Send every invoice carrying a deposit, a credit note or a discount to a human, always. That is where 100 percent of the errors in this test landed, and it is the one place where the arithmetic check is simultaneously useless and misleading. Detecting those documents does not need a model: the words are in the text.

What you can stop doing is spending effort on the layout of the PDF before extraction. On this corpus it bought nothing.

If you want the broader version of this reasoning, we wrote about the tasks where AI fails without saying so and about the failure modes of automated workflows. This test is one measured instance of both.

Limits, stated plainly

• 24 synthetic invoices. A test on 24 cases is a test on 24 cases.
• One run per combination, at temperature 0. Models are not perfectly deterministic even so. Gaps of one or two points between models should not be read as significant. Gaps of ten points and more, and above all the structure of the errors, always the same field and the same kind of invoice, are.
• Text input, not images. A real scan adds character errors this test does not measure, and they can only make the result worse.
• English only.
• Three entry level models. Nothing here says what the more expensive ones would do. That is an open question and it deserves its own test.
• The corpus is ours. It is published so that disagreement is possible.

Rerun it

Everything this test produced is here, not summarised: the scripts that generated the corpus and degraded it, the 24 invoices with their ground truth, the 288 raw responses exactly as the models returned them, and the script that turns those into the numbers above.

Seven files, 393777 bytes: the corpus generator with its fixed seed, the 24 invoices with their ground truth, the four degradation levels, the script that called the models and compared field by field, the 288 raw responses with the prompt used, and the analysis that produces every figure above. The data page lists them one by one with what each holds.

The generator uses a fixed seed, so the corpus you produce is the corpus we used.

python3 generer-corpus.py --sortie corpus.json
python3 conduire-test.py --corpus corpus.json --sortie resultats-bruts.json \
    --niveaux 1 2 3 4 \
    --modeles openai/gpt-4o-mini google/gemini-2.5-flash-lite anthropic/claude-haiku-4.5
python3 analyser.py --resultats resultats-bruts.json --sortie chiffres.json

The API key is read from an environment variable and appears nowhere in these files.

Download the dataset (zip, 27.8 KiB, 7 files, with the full inventory and the checksum). The scripts and the comments inside them are in French, which is the working language behind this site.

If you rerun this and get something different, we want to know. There is a contact form, and corrections get published on the article itself, dated, as the editorial rules of this site say.