Does paying more buy a better invoice extraction? 864 calls, six models, the real bill

Three low cost models already read a supplier invoice and get 81 to 89 % of them entirely right. The obvious next question, for anyone about to put this in front of their accounts payable, is whether the expensive model from the same vendor fixes the rest, and what that costs per thousand invoices.

We ran it. 864 calls, six models, three vendors, 2.02 USD of real billing. Here is what came back, before the method.

On clean invoices the upgrade buys nothing at all. At degradation levels 1 and 2, which is 288 of the 576 extractions, all six models are at 100 %. Every point of difference in this test comes from the two levels where the document itself is damaged.

Where it does buy something, all three vendors buy the same amount. Moving up a tier gains 10, 11 and 11 correct invoices out of 96. Three different vendors, three different price ratios, one identical gain.

And the price of that identical gain varies by a factor of 6.5. The extra correct invoice costs 0.0153 USD with OpenAI, 0.0364 USD with Anthropic, and 0.1001 USD with Google. Same result, six and a half times the bill.

The model that scores 100 % is also the one that flags 82 % of its own correct work. google/gemini-2.5-pro got all 96 invoices right and wrote a warning note on 79 of them. Its note carries no information at all, and reading it as a quality signal would send four out of five clean invoices to a human.

Method, in short

Test conducted on 2026-09-06 by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, published below so that anyone can rerun them and disagree with the result. Etienne signs this article and answers for its content.

This is the sixth measured test published here, and the second that puts a price on an accuracy figure rather than reporting the figure alone.

The corpus was not regenerated. It is exactly the one published on 2026-08-13 under the invoice extraction dataset: 24 script generated supplier invoices, seed 20260810, four levels of text degradation, seven fields per invoice, the same instruction word for word, temperature 0. Regenerating it would have mixed the effect of price with the effect of a different corpus, and no figure below would separate the two.

The six models are three pairs. Same vendor, same generation, only the tier changes.

That pairing is deliberate. Comparing a small 2024 model to a large 2026 one would measure the generation as much as the tier.

The price ratio that matters is not the advertised one. Every cost below is the amount actually billed for the call, read out of the API response, never estimated from a price list. A reasoning model pays for its reasoning tokens too, and that is where the advertised ratio and the real one part company.

Two passes, and why there had to be two

Pass A, output budget 700 tokens, the three economy models, 288 calls. That is the 2026-08-10 protocol to the token. It is not there to compare prices: it is a drift witness, and it answers the question the 2026-09-04 protocol explicitly ruled out of its own scope.

Pass B, output budget 4000 tokens, six models, 576 calls. A 700 token budget truncates gemini-2.5-pro, which spends 600 to 1450 reasoning tokens before answering. Measuring it at 700 would have produced a false accuracy result where there is only a budget that is too short. The accuracy comparison is therefore read on pass B, where no model is handicapped.

Control. On the three economy models, which do not reason, the two budgets return the same value on 285 of 288 cases, 99.0 %. The budget is not a hidden variable of the comparison.

An incident during the measurement, and what it cost

On the first run of pass B, 81 of the 96 claude-opus-4.5 calls failed with HTTP 402: the remaining OpenRouter credit did not cover the calls in flight at 8 concurrent requests. The harness inherited from 2026-08-10 counts a technical failure as a wrong extraction, so the table showed 4/24, 2/24, 7/24, 2/24 for that model. Left alone, this article would have said that the most expensive Anthropic model is the worst of the six.

Those 96 calls were replayed alone, at 3 concurrent requests, same corpus, same instruction, same budget. Zero technical failures, zero invalid JSON, and the model returns 89 of 96. Both files are in the archive: the failed run kept as it is, and the merged valid run.

Two lessons, written down because they will serve again. A harness that confuses "the model got it wrong" with "the call did not go through" manufactures results that are false and plausible. And a spectacular gap in the counter-intuitive direction gets diagnosed before it gets written, never after.

Raw figures

Accuracy is the share of invoices whose seven fields are all correct, out of 96 cases per model, 24 invoices at four degradation levels. Cost is dollars actually billed, scaled to 1000 invoices.

What the upgrade buys, pair by pair

All three pairs go the same way, and the size of the gain barely moves: 10 or 11 invoices out of 96. What moves is the bill. The advertised price ratios were 16.7x, 12.5x and 5x; the ratios actually billed are 17.1x, 121.7x and 4.9x. Google's premium model costs ten times what its price list suggests, because it pays for 956 reasoning tokens per invoice on top of its answer.

Two cases out of 288 where the premium model fails and the economy model succeeds, both at OpenAI: F19 at level 4 and F24 at level 3. gemini-2.5-pro and claude-opus-4.5 regress on none of their junior's cases.

By degradation level, and this is the row that decides

Accuracy per model, per level, 24 invoices each.

Read the first two rows. On 288 of the 576 extractions, the cheapest model in the test and the most expensive one return exactly the same thing: everything correct. If your invoices arrive as clean text, the upgrade buys you nothing, and this table is the reason to spend the money on your input pipeline instead of your model tier.

The whole difference lives at N3 and N4, where the layout is broken and the labels are gone. That is also where the spread between models is widest: at N4, gpt-4o-mini drops to 58.3 % while gpt-4o holds 95.8 %.

The model's own note is not an error detector

Every model was offered a free text field to flag anything it was unsure about. Catch rate is the share of its wrong extractions it flagged. False alarm is the share of its correct ones it flagged.

This is the result that must not be read sideways. Four models flag 100 % of their errors, which looks excellent, and flag 34 to 49 % of their correct work at the same time. Precision falls to 14 to 37 %: of ten extractions the model flags, seven to nine are right.

gemini-2.5-pro is the clearest case. It made zero errors and wrote a note on 79 of its 96 correct extractions. There is no error for that note to point at. Trusting it would mean re-reading 82 % of a perfect batch.

At the opposite end, gpt-4o-mini never writes a note, on any of the 96. Its 13 errors are therefore all silent, and it is not hiding them: it does not use the field.

Everything here is consistent with what the 2026-09-04 test found on the checks placed around a model: a catch rate published without its false alarm rate is a number that decides nothing.

Drift at 27 days, same corpus, same instruction

Pass A against the raw results of 2026-08-10, case by case, on the same 288 calls.

The six cases that differ all concern total_due or subtotal, and exactly one flips from correct to wrong: gpt-4o-mini, invoice F08, level 3. At temperature 0, over a month, these three models return very nearly the same thing, and their errors are stable errors, not noise.

That matters beyond this test. It means the August dataset is still a fair description of what those models do today, and it is the reason the guardrail comparison of 2026-09-04 could reuse it. It does not mean temperature 0 is repeatable within a single day: we measured that separately and it is not.

The instruction, unchanged since August

# La consigne envoyee aux six modeles, identique a celle du 2026-08-10.
CONSIGNE = """You are extracting data from a supplier invoice for a small business bookkeeping system.

Return ONLY a JSON object with exactly these keys:
- invoice_number: string, the supplier's invoice number
- invoice_date: string, ISO format YYYY-MM-DD
- due_date: string, ISO format YYYY-MM-DD
- currency: string, ISO 4217 code
- subtotal: number, the amount the tax was calculated on
- tax: number
- total_due: number, the amount the customer must actually pay for this invoice
- notes: string, empty if everything was unambiguous, otherwise say what was unclear

Numbers must be plain numbers, no currency symbols, no thousand separators.

Invoice:
---
%s
---"""

# Le cout est LU dans la reponse de l'API, jamais estime depuis une grille
# tarifaire : usage.cost porte le montant reellement facture, jetons de
# raisonnement compris, et c'est la seule mesure de prix de ce test.
charge = {"model": modele, "messages": [{"role": "user", "content": CONSIGNE % texte}],
          "temperature": 0, "max_tokens": budget_sortie,
          "usage": {"include": True}}
...
usage = donnees.get("usage") or {}
return {"jetons_raisonnement": ((usage.get("completion_tokens_details") or {})
                                .get("reasoning_tokens")),
        "cout_usd": usage.get("cost")}

What this does not prove

• 24 script generated invoices are not a sample of real invoices. No scan, no foreign language, no exotic layout, no handwritten stamp.
• One call per case. The accuracy rates carry a sampling uncertainty that is not measured here, and the 2026-08-25 test established that temperature 0 does not guarantee repeatability.
• API prices change. The costs published are those of 2026-09-06.
• Nothing here says a premium model stays better on another task. It says what it did on this one.
• The 100 % of gemini-2.5-pro is 96 out of 96. A perfect score on 96 cases is a perfect score on 96 cases, not a guarantee.

What we would actually do with this

If invoices arrive as clean text, stay on the economy model: on this corpus it is already at 100 % and the upgrade is pure cost. If they arrive damaged, the upgrade is worth it, and the vendor you pick decides whether the same gain costs you 15 USD or 100 USD per thousand invoices. And in both cases, do not use the model's own note as your check: the checks that work are measured here, and the model's spontaneous note is the weakest of them.

The full corpus, the 864 raw responses, the two passes including the failed one, and the analysis scripts are on the dataset page for this test.

One warning about the premium tier, measured since: on a task where the model has to decide rather than read, the same upgrade buys fewer false holds and more wrong approvals. We measured it on invoice against purchase order matching, and on an accounts payable process that is the wrong trade.