Which check actually catches a bad extraction? Four guardrails, 288 extractions

A model reads an invoice, returns clean JSON, and gets a number wrong. Nothing crashes. The test we published on 2026-08-13 measured how often that happens and left a figure hanging that nobody picked up: the arithmetic check meant to catch those errors also fired on 25 % of the extractions that were perfectly correct.

A check that cries wolf a quarter of the time does not go into production. So we took the same 288 extractions and put four different guardrails on them. Here is what came back, before the method.

The guardrail that catches the most is the worst one to deploy. Asking the model to check its own work catches 41 of the 42 wrong extractions, 97.6 %. It also flags 39 % of the correct ones. Per 1000 invoices you would read 476 of them by hand to find 142 errors.

Cross-model agreement catches almost as much and costs a third of the reading. Run three models, flag any invoice where they do not all agree: 92.9 % of the errors caught, 11 % false alarms, 229 invoices to re-read per 1000 instead of 476.

On clean invoices, cross-model agreement never fires. Zero false alarms at degradation levels 1 and 2, on 144 correct extractions. The arithmetic check fires on 25 % of those same invoices. That difference is the whole finding.

When the self-check flags a correct extraction, 30 % of the time it asks for the value that is already there. Not a paraphrase of that claim, the literal thing: subtotal should be 1581.38, but it is incorrectly extracted as 1581.38.

Method, in short

Test conducted on 2026-09-04 by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, published below so that anyone can rerun them and disagree with the result. Etienne signs this article and answers for its content.

This is the fifth measured test published here, and the first that does not measure a model. It measures the checks you would put around one.

The 288 extractions were not redone. They are exactly the ones published on 2026-08-13 under the invoice extraction dataset: 24 script generated supplier invoices, seed 20260810, four levels of text degradation, three low cost models. 42 of those 288 extractions are wrong, 246 are right, and which is which was settled then against the generated ground truth.

That choice is deliberate and it is the reason the comparison means anything. All four guardrails judge an identical set of extractions. Rerunning the models today would have mixed the effect of the guardrails with 25 days of model drift, and no figure below would separate the two.

What it costs to say that honestly: nothing here tells you whether those same models still make the same errors today. This test is about the checks, not about the models. The other half of that question was measured on 2026-09-06: whether a premium model makes fewer errors, and at what price.

The four guardrails

Two of them are arithmetic and cost nothing. One costs a call per invoice. One costs extra extractions.

def g1_arithmetique_bilaterale(r):
    # Le controle publie le 2026-08-13, repris tel quel comme temoin.
    return abs(r["total_due"] - (r["subtotal"] + r["tax"])) >= 0.011

def g2_arithmetique_unilaterale(r):
    # Ne signale que si on reclame PLUS que ce que l'arithmetique porte.
    # Un acompte deja verse fait legitimement baisser le total, jamais monter.
    return r["total_due"] - (r["subtotal"] + r["tax"]) >= 0.011

def g3_auto_verification(facture, r):
    # Un appel de plus, au MEME modele, avec la facture et sa propre sortie.
    # La consigne ne dit jamais qu'une erreur a ete trouvee.
    return not appeler(modele, CONSIGNE % (facture, r))["correct"]

def g4_accord_entre_modeles(r, autres):
    # Unanimite exigee sur les 7 champs. Un desaccord, on releve la facture.
    return any(r[c] != autre[c] for autre in autres for c in CHAMPS)

G2 is there because of a hypothesis we had before running anything: the 25 % false alarm rate of G1 might be rounding noise, and a one sided check would remove it. That hypothesis is wrong, and the section below says why.

Raw figures

288 extractions judged, 42 of them wrong. Catch rate is the share of the 42 that a guardrail flags. False alarm is the share of the 246 correct ones it flags. Both numbers, always, because either one alone can be made perfect by a guardrail that is useless.

What each one costs you in reading

The number that decides whether a check ships is not its catch rate. It is how many documents a person has to open.

G4 finds 135 of the 146 errors in a thousand invoices and sends 229 invoices to a human. G3 finds 142, seven more, and sends 476. Seven extra errors for 247 extra documents to read is not a trade a bookkeeping team makes.

By degradation level, where it gets interesting

Read the first two rows. There are no errors to catch at levels 1 and 2: all 144 extractions are correct. The arithmetic check flags 36 of them. The self-check flags 49. Cross-model agreement flags none.

A real invoice stream is mostly clean documents. On those, G1 and G3 generate pure noise and G4 generates silence. That is what separates a check you keep from a check your team turns off in the second week.

Why the one sided arithmetic check catches nothing

The hypothesis was that a legitimate deposit makes total_due smaller than subtotal + tax, while an extraction error could go either way, so ignoring the negative side would remove the false alarms and keep the catches.

Here is the direction of every arithmetic gap in the 288 extractions.

Not one gap in 288 is positive. Errors make the total too low, which is exactly what a deposit already paid does legitimately. The direction of the discrepancy carries no information at all, so the one sided check has nothing to fire on and catches 0 of 42.

That also explains the 25 % on clean invoices: those 54 negative gaps on correct extractions are not rounding, they are deposits. The arithmetic identity is simply false for that kind of invoice, and no tolerance fixes it. It is the wrong check, not a badly tuned one.

The self-check argues with itself

G3 has the best catch rate in the table, and reading its output is what stops you from shipping it.

When a model flags an extraction, it also returns a problems string. On the correct extractions it flagged, 40 of those strings name a field and a value. 12 of those 40, exactly 30 %, ask for the number that is already in the extraction.

extraction : subtotal 1581.38, tax 79.07, total_due 1660.45   (all correct)
verdict    : correct = false
reason     : subtotal should be 1581.38, but it is incorrectly
             extracted as 1581.38; tax should be 79.07, but it is
             incorrectly extracted as 79.07; total_due should be
             1660.45, but it is incorrectly extracted as 1660.45

That is openai/gpt-4o-mini on invoice F01, level 2, checking its own output. It re-read the invoice, derived the right values, found them identical to what it was given, and returned correct: false anyway.

A 97.6 % catch rate built partly on this is not detection. It is a model that leans towards saying no when asked whether something is wrong, and it happens to be right often because there are errors to be found. The precision figure says it plainly: of everything G3 flags, 29.9 % is actually wrong.

We did not tune the prompt to fix this. The instruction is the plain one a small business would write, it never hints that an error exists, and it is published below. Making it stricter would measure our prompt engineering, not the guardrail.

One extra model is enough, and it is the cheapest line in the table

G4 needs three models. Most teams will run two. So we measured every pair separately: keep your current model, add one, flag any invoice where the two disagree on any field.

A second opinion from gpt-4o-mini catches as many errors as the arithmetic check, 59.5 % against 59.5 %, with 5.3 % false alarms instead of 22.0 %. Same catch, a quarter of the noise, for the cost of one extra cheap call.

The third row is the honest one. Adding claude-haiku-4.5 as the second reader catches 26.2 %. The value of a second model depends on which model, and this test cannot tell you why. What it can tell you is that the choice matters more than the idea.

Costs, measured not estimated

Per 1000 invoices, at the prices billed during this run.

The self-check pass, 288 calls, cost 0.086 USD in total. None of these guardrails is expensive. The expensive resource is the person who reads the flagged invoices, which is why the reading column above is the one that decides.

What we would do with this

Do not check arithmetic on invoices that can carry a deposit. The identity subtotal + tax = total_due is false for a whole legitimate category of documents, and no tolerance repairs that. On this corpus it costs 36 false alarms on 144 clean invoices and buys 25 catches out of 42.

Add one cheap second model and compare the fields. It is the best ratio in this test: 1.5 documents read per error found, 5.3 % false alarms, one extra call. Start there before anything more elaborate.

Do not ask a model to grade its own output and act on the verdict. Read the reasons first. On this corpus, 30 % of the reasons given for flagging a correct extraction ask to replace a value with itself. The verdict looked usable; the reasoning was not.

Measure both numbers, always. Any single one of these guardrails can be made to look excellent by quoting only its catch rate or only its false alarm rate. G2 has a perfect 0 % false alarm rate and catches nothing.

Limits, stated plainly

This section is not a formality. It is the condition for the rest to be readable.

• The extractions are 25 days old. They were produced on 2026-08-10. Nothing here says the same models make the same errors today, and this test was not designed to find out.
• 42 errors. Every catch rate above is computed on 42 wrong extractions. A percentage on 42 items has a wide margin. These are orders of magnitude, not constants.
• One corpus, 24 constructed invoices, fixed seed, declared as constructed in its own protocol. No real supplier invoice. The deposit trap that breaks the arithmetic check is a designed feature of that corpus, and its frequency there is not the frequency in your ledger.
• Three models, one provider, one day. The self-check pass went through OpenRouter on 2026-09-04.
• G4 measured with three models is not what most teams will run. That is why the pairwise table is there, and the pairwise numbers are lower.
• The self-check prompt is one prompt. A different instruction would give different numbers. We report the one we ran and publish it rather than searching for a flattering variant.
• Nothing here measures the cost of a missed error. All the trade-offs above weigh reading time against catches. An invoice paid twice does not cost the same as an invoice read twice, and that ratio is yours, not ours.

Rerun it

Everything is published, not summarised: the 288 extractions being judged, the self-check script, the 288 raw verification responses exactly as the models returned them, and the analysis that produces every figure above.

python3 auto-verifier.py --extractions extractions.json --corpus corpus.json \
    --sortie verifications.json
python3 analyser.py --extractions extractions.json \
    --verifications verifications.json --sortie chiffres.json

The analysis calls no model, so it replays at zero cost, and a guardrail rule you disagree with can be changed and rerun on the same 288 responses. The API key is read from an environment variable and appears nowhere in these files.

Download the dataset (zip, with the full inventory and the checksum). The scripts and the comments inside them are in French, which is the working language behind this site.

If you rerun this and get something different, we want to know. There is a contact form, and corrections get published on the article itself, dated, as the editorial rules of this site say.

These guardrails check a reading. A different kind of check is needed once the model has to compare the invoice with the purchase order, and there one discrepancy out of seven gets through every time: the one the instruction itself taught the model to forgive.