THINK AI FIRST

Dataset

Four error detection guardrails on 288 invoice extractions

288 invoice extractions already published here, four checks applied to every one of them, and the 288 raw responses of the check that needed model calls. Test conducted on 2026-09-04.

This page describes one dataset, published in full. It is the raw material behind our comparison of four error detection guardrails: 288 invoice extractions already published here, four different checks applied to every one of them, and the 288 raw responses of the check that needed model calls.

Everything on this page comes from that one run. Nothing is estimated.

What it measures, and what it does not prove

It measures no model. It measures the checks you would put around one to catch a wrong extraction before it reaches a bookkeeping system.

Four guardrails are applied to the same 288 extractions, and each is scored on two quantities that are never read one without the other: the catch rate, the share of the wrong extractions it flags, and the false alarm rate, the share of the correct ones it flags. A guardrail that flags everything scores 100 % on the first and is useless. One that flags nothing scores 0 % on the second and is useless too.

What it does not prove, stated plainly, because a dataset quoted without its limits does more harm than no dataset at all:

  • The extractions are 25 days old. They were produced on 2026-08-10. Nothing here says the same models make the same errors today, and this test was not designed to find out.
  • 42 errors. Every catch rate rests on 42 wrong extractions. A percentage on 42 items has a wide margin. These are orders of magnitude, not constants.
  • One corpus, 24 constructed invoices, fixed seed, declared as constructed in its own protocol. No real supplier invoice. The deposit trap that defeats the arithmetic check is a designed feature of that corpus, and its frequency here is not its frequency in a real ledger.
  • Three models, one provider, one day. The self-check pass went through OpenRouter on 2026-09-04.
  • The self-check prompt is one prompt. A different instruction would give different numbers. The one that was run is published inside verifications.json, rather than a flattering variant found afterwards.
  • Nothing measures the cost of a missed error. Every trade-off here weighs reading time against catches.

The extractions are reused, and here is what that means

Nothing was extracted for this test. The 288 extractions judged here are exactly those published on 2026-08-13 under the invoice extraction dataset: 24 script generated supplier invoices from seed 20260810, four levels of text degradation, three low cost models, seven fields per invoice compared against the generated ground truth. 42 of them are wrong, 246 are right, and that split was settled then.

That is a deliberate trade and it cuts both ways. What it buys: all four guardrails judge an identical set of extractions, so the comparison isolates the guardrails and nothing else. What it costs: rerunning the models on 2026-09-04 would have mixed their effect with 25 days of model drift, and this dataset says nothing about the models as they are today.

This dataset generates no corpus of its own. Its reproducibility rests on the seed of the source corpus, 20260810. Regenerate it and you hold the same 24 invoices.

Test conducted on 2026-09-04. The calls were run by the automated testing pipeline of this site: the self-check pass and the analysis are scripts, and both are in the archive below.

The guardrails

What it does Cost
G0 the model's own notes field, filled during the original extraction none
G1 arithmetic, both directions: |total_due - (subtotal + tax)| >= 0.011 none
G2 arithmetic, one direction only: flags when more is claimed than the arithmetic carries none
G3 self-check: one extra call to the same model, given the invoice and its own output one call per invoice
G4 agreement of three models: any field disagreement flags the invoice two extra extractions
G5 agreement with a single other model, computed for all three pairs one extra extraction

The G3 instruction never hints that an error was found and gives no ground truth. Telling the model an error exists would measure compliance, not detection.

Raw figures

288 extractions judged, 42 wrong, 246 right.

Guardrail Catch rate False alarms Precision Errors missed
G0 spontaneous note 69.0 % 24.4 % 32.6 % 13
G1 arithmetic, both directions 59.5 % 22.0 % 31.6 % 17
G2 arithmetic, one direction 0.0 % 0.0 % n/a 42
G3 self-check 97.6 % 39.0 % 29.9 % 1
G4 agreement of three models 92.9 % 11.0 % 59.1 % 3
G3 and G4 together 97.6 % 41.1 % 28.9 % 1
G5 with gpt-4o-mini alone 59.5 % 5.3 % 65.8 % 17
G5 with gemini-2.5-flash-lite alone 47.6 % 2.8 % 74.1 % 22
G5 with claude-haiku-4.5 alone 26.2 % 7.3 % 37.9 % 31

Reading load per 1000 invoices

A thousand invoices at this error rate carry 146 errors.

Guardrail To re-read Errors found Errors missed Reads per error
G1 274 87 59 3.2
G3 476 142 4 3.3
G4 229 135 10 1.7
G5 with gpt-4o-mini 132 87 59 1.5

False alarms by degradation level

Level Wrong extractions G1 G3 G4
N1 clean 0 25.0 % 30.6 % 0.0 %
N2 columns lost 0 25.0 % 37.5 % 0.0 %
N3 totals scrambled 21 11.8 % 47.1 % 29.4 %
N4 labels lost 21 23.5 % 45.1 % 23.5 %

At levels 1 and 2 there is no error to catch: all 144 extractions are correct. G1 flags 36 of them, G3 flags 49, G4 flags none.

Direction of the arithmetic gap

Gap negative Exact Gap positive
Wrong extractions (42) 25 17 0
Correct extractions (246) 54 192 0

Not one gap in 288 is positive, which is why G2 catches nothing: an error makes the total too low, exactly as a deposit already paid does legitimately.

What the G3 reasons are worth

Auditable reasons Ask for the value already present Share
On correct extractions 40 12 30.0 %
On wrong extractions 25 4 16.0 %

Costs

The self-check pass, 288 calls: 0.08638 USD, 288 read, 288 valid JSON, 0 technical failures. Per 1000 invoices: G0, G1 and G2 cost nothing, G3 costs 0.300 USD, G4 costs 0.830 USD, G5 costs 0.415 USD.

What the archive contains

Eight files, 566833 bytes uncompressed. The inventory is here so you know what you are downloading before you download it.

File What it holds Size
corpus.json the 24 invoices and their ground truth, seed 20260810 28.6 KiB
degrader.py the four degradation levels, needed to hand the model the exact text it saw 2.6 KiB
extractions.json the 288 extractions being judged, each with its per field correctness 307.8 KiB
auto-verifier.py the G3 pass, one call per extraction 5.3 KiB
verifications.json the 288 raw self-check responses, with the instruction used 135.5 KiB
analyser.py the four guardrails, the reading load, the audit of the G3 reasons 12.9 KiB
chiffres.json every computed figure 49.2 KiB
PROTOCOLE.md the protocol, the reading of the figures, and the limits 11.6 KiB

corpus.json and degrader.py are copied unchanged from the 2026-08-13 dataset, so this archive stands on its own without fetching another one.

The scripts and their comments are in French, which is the working language behind this site. The API key is read from an environment variable and appears in none of these files.

Rerun it

The analysis calls no model at all, so it replays at zero cost on the raw responses shipped in the archive. Only auto-verifier.py spends anything.

bash
unzip 2026-09-extraction-guardrails.zip
python3 auto-verifier.py --extractions extractions.json --corpus corpus.json \
    --sortie verifications.json
python3 analyser.py --extractions extractions.json \
    --verifications verifications.json --sortie chiffres.json

A guardrail rule you disagree with can be changed in analyser.py and rerun on the same 288 responses without paying for a single call.

Each raw response is one entry of verifications.json, exactly as the model returned it, before any scoring. This one is the self-check that flagged a correct extraction and asked for the values already in it:

json
{"modele": "openai/gpt-4o-mini", "facture": "F01", "niveau": 2, "lu": true,
 "json_valide": true, "dit_correct": false,
 "problemes": "subtotal should be 1581.38, but it is incorrectly extracted as 1581.38; tax should be 79.07, but it is incorrectly extracted as 79.07; total_due should be 1660.45, but it is incorrectly extracted as 1660.45"}

Verify what you downloaded before you trust it:

bash
shasum -a 256 2026-09-extraction-guardrails.zip
# 3438d27888762a3b2d0d5794d214f95bfd9a1692a40b8507353e62074e0ff754

How to cite this dataset

Copy one of the two forms below. The reference points at this page, which is the stable address of this dataset; the archive is what that address serves.

citation
Think AI First (2026). Four error detection guardrails on 288 invoice extractions, 288 raw self-check responses.
Think AI First. Test conducted 2026-09-04, published 2026-09-04.
https://think-ai-first.com/datasets/extraction-guardrails-2026-09. Licensed CC BY 4.0.
bibtex
@dataset{thinkaifirst2026guardrails,
  author    = {{Think AI First}},
  title     = {Four error detection guardrails on 288 invoice extractions, 288 raw self-check responses},
  year      = {2026},
  publisher = {Think AI First},
  url       = {https://think-ai-first.com/datasets/extraction-guardrails-2026-09},
  note      = {Test conducted 2026-09-04. Licensed CC BY 4.0. Archive SHA-256 3438d27888762a3b2d0d5794d214f95bfd9a1692a40b8507353e62074e0ff754}
}

Licence and reuse

Published under CC BY 4.0. Use it, quote it, contradict it. If you rerun this and get something different, tell us through the contact form: corrections are published on the article itself, dated.

Download the dataset (zip, 44.4 KiB, 8 files)

Other datasets published here

Each one is a separate test, with its own corpus, its own raw responses and its own archive. This one judges the extractions of the first.