Four models catch every blunt invoice error, and approve the over billing anyway. 672 calls

Ask a model to check a supplier invoice against its purchase order and it will catch a wrong unit price, a line that was never ordered, a purchase order number off by one digit and a currency that does not match. Every single time. 383 cases out of 383, four models, no exception.

Then it will approve an invoice for three units when you ordered one, and write in its own answer that three were invoiced and one was ordered.

We ran it. 672 calls, four models, 168 invoice and purchase order pairs, 0.4672 USD of real billing. Here is what came back, before the method.

The four blunt discrepancies are caught 100 % of the time. Wrong unit price, line absent from the order, wrong order reference, wrong currency: 383 exploitable cases, 383 correct decisions, across all four models. There is nothing to fix there.

All seventeen wrong approvals of the test sit on one single condition. Over billing: more units invoiced than ordered. Zero wrong approvals on the six other conditions. Not one.

The cause is written in the models' own words. Our instruction says that partial delivery is acceptable, because it is. The models apply that exemption without checking which way the quantity moved. gpt-4o wrote Packing tape, 48mm, pack of 6: invoiced quantity is 3, ordered quantity is 1 in its list of discrepancies, then concluded: The invoice matches the purchase order except for a higher quantity of packing tape, which is acceptable.

And the same clause fails in the other direction too. A genuine partial delivery, fewer units at the agreed price, is approved only 33 times out of 96. Two correct invoices out of three are held for a human who has nothing to decide.

The premium model is the best overall and the worst where it costs money. gpt-4o gets 154 of 168 right, against 128, 134 and 141 for the three economy models. It also commits 8 wrong approvals against 2, 3 and 4. What it buys is a collapse in false holds, from 23 to 37 down to 6. What it sells you in exchange is over billing that goes through.

Method, in short

Test conducted on 2026-09-14 by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, published below so that anyone can rerun them and disagree with the result. Etienne signs this article and answers for its content.

This is the eighth measured test published here, and the first where the model has to compare two documents instead of reading one.

The invoices were not regenerated. They are exactly the 24 published on 2026-08-13 under the invoice extraction dataset, generated by script with the fixed seed 20260810. What is new is the purchase order paired with each of them.

Each invoice gets seven purchase orders, one per condition, carrying one discrepancy and one only. Two of the seven are legitimate approvals: without them, a model that refuses everything would score perfectly.

C3 shows 95 and not 96 because one call out of 672 came back with truncated JSON. It is counted neither right nor wrong.

The four models, and what each decision costs

Every cost here is the amount actually billed for the call, read out of the API response, never estimated from a price list.

Where the premium tier actually changes something

Read the last row against the two above it. The tier that learns to stop refusing correct invoices is the same tier that starts accepting over billing. On this task, those two are not independent.

The instruction, word for word

# La consigne envoyee aux quatre modeles, identique pour tous.
# La clause sur la livraison partielle est celle qui produit les 17 faux
# approve : les modeles l'appliquent sans verifier le SENS de l'ecart.
CONSIGNE = """You are checking a supplier invoice against the purchase order it refers to, for a small business accounts payable process.

Approve the invoice only if it is consistent with the purchase order. Partial delivery is normal and acceptable: fewer units than ordered, at the agreed unit price, is not a reason to hold.

Return ONLY a JSON object with exactly these keys:
- decision: string, either "APPROVE" or "HOLD"
- reason: string, one sentence
- discrepancies: array of strings, every difference you found between the invoice and the purchase order, empty if none

Purchase order:
---
%s
---

Invoice:
---
%s
---"""

The discrepancies field is not decoration. It is what separates a model that did not see the discrepancy from one that saw it and approved anyway. Of the 17 wrong approvals, 16 came with an empty list: those models did not read the quantity. One, the gpt-4o case quoted above, listed the discrepancy and approved regardless. The first is a reading problem. The second is a judgement problem, and no bigger model fixes it: only an explicit rule does.

What this does not prove

• 24 script generated invoices are not a sample of real invoices. No scan, no foreign language, no exotic layout, no handwritten stamp.
• One discrepancy at a time. Real invoices combine them, and nothing here says the models behave the same way when two problems overlap.
• One purchase order format. A different layout may well move these numbers.
• Text, not PDF. The models were given plain text, not a scanned document.
• One call per case, at temperature 0, and our own test of 2026-08-25 established that temperature 0 does not guarantee repeatability.
• A known flaw in the corpus, measured rather than hidden. Six of the 24 invoices carry two lines with the same description at different unit prices, which makes their table ambiguous before any discrepancy is introduced. Every figure per condition is therefore published twice, with and without those six. The gap is small: C4 goes from 34.38 % to 36.11 %, C0 from 64.58 % to 63.89 %. The duplication does not explain the result.

What we would do with this

Not "use a better model". The premium tier bought us fewer false holds and more over billing that went through, which on an accounts payable process is the wrong trade.

What the numbers point at is narrower and cheaper: the quantity comparison should not be left to the model's judgement at all. Two numbers, one subtraction, one sign. Compute it in code, hand the model the sign, and the 17 wrong approvals of this test disappear without changing the model or its price. Everything the models do well here, reading two documents and matching lines that are worded differently, they keep doing.

Download the dataset: the 168 pairs, the 672 raw answers, the generator, the harness and the analysis script, with the archive and its checksum.