THINK AI FIRST

Dataset

Invoice against purchase order matching, four models

The 24 invoices already published here, each paired with seven purchase orders carrying one controlled discrepancy and one only, judged by four models across 672 calls. Test conducted on 2026-09-14.

This page describes one dataset, published in full. It is the raw material behind our measurement of invoice against purchase order matching: 168 pairs of documents, 672 decisions from four models, the discrepancies each model listed, and the cost billed for every call.

Everything on this page comes from that one run. Nothing is estimated.

What it measures, and what it does not prove

It measures one thing: which discrepancies between an invoice and its purchase order a model catches, and which one it lets through. Seven conditions, one discrepancy at a time, two of them legitimate approvals so that a model refusing everything cannot score well.

This is the first dataset published here where the model compares two documents. The seven before it all handed the model a single document to read.

What it does not prove, stated plainly, because a dataset quoted without its limits does more harm than no dataset at all:

  • 24 script generated invoices are not a sample of real invoices. No scan, no foreign language, no exotic layout, no handwritten stamp.
  • One discrepancy at a time. Real invoices combine them.
  • One purchase order format, generated by our own script.
  • Text, not PDF. No optical character recognition is involved anywhere.
  • One call per pair, at temperature 0, and our own test of 2026-08-25 established that temperature 0 does not guarantee repeatability.
  • Six invoices out of 24 carry two lines with the same description at different unit prices, which makes their table ambiguous before any discrepancy is introduced. Every per condition figure is therefore published twice, with and without those six, in chiffres.json.

The invoices are reused, the purchase orders are new

The 24 invoices and their ground truth are exactly those published here on 2026-08-13 under the invoice extraction dataset, generated by script with the fixed seed 20260810. They were not regenerated.

What is new is the purchase order paired with each of them. generer-bons-de-commande.py reads each invoice table back out of the document, then writes seven purchase orders for it. The random draw is seeded per invoice, so the same invoice always gets the same discrepancy and no two invoices get the same one.

The seven conditions

Code Discrepancy introduced in the purchase order Correct answer
C0 none, the invoice matches the order APPROVE
C1 more units invoiced than ordered HOLD
C2 unit price invoiced above the price ordered HOLD
C3 an invoiced line is not on the order HOLD
C4 partial delivery: fewer units, correct prices APPROVE
C5 order reference differs by one digit HOLD
C6 order currency differs from the invoice HOLD

C4 is the condition that separates a model which understands from one that refuses as soon as a number differs.

What is in the archive

File What it holds
PROTOCOLE.md The question, the sub intention it targets, the seven conditions, what the test does not prove. Written before the test was run.
generer-bons-de-commande.py Builds the 168 pairs from the August corpus. No network, no ungrounded randomness.
paires.json The 168 pairs, each with its invoice text, its purchase order text, its condition and the correct decision.
conduire-test.py The harness: one call per pair and model, temperature 0, the instruction word for word.
resultats-bruts.json The 672 raw answers, with the decision, the reason, the discrepancies listed, the finish reason, the token counts and the billed cost.
analyser.py Turns the raw answers into the published figures, and nothing else.
chiffres.json The published figures, per model and per condition, each with and without the six ambiguous invoices.
factures-a-description-dupliquee.json The six invoice identifiers whose table is ambiguous before any discrepancy.

One raw record, as it is served

Every line of resultats-bruts.json holds one decision with what the model said about it. This one is the case quoted in the article: the model wrote the discrepancy down in ecarts_vus, then approved anyway.

json
{
 "modele": "openai/gpt-4o",
 "paire": "F18-C1",
 "condition": "C1",
 "attendu": "HOLD",
 "lu": true,
 "json_valide": true,
 "decision": "APPROVE",
 "juste": false,
 "motif": "The invoice matches the purchase order except for a higher quantity of packing tape, which is acceptable.",
 "ecarts_vus": ["Packing tape, 48mm, pack of 6: invoiced quantity is 3, ordered quantity is 1"]
}

Verify what you downloaded before you trust it:

bash
shasum -a 256 2026-09-invoice-po-matching.zip
# 559ce436f28746c1baaa4974eef404da61046ed7e9e3ebdd2bb8272b4824c314

How to cite this dataset

Copy one of the two forms below. The reference points at this page, which is the stable address of this dataset; the archive is what that address serves.

citation
Think AI First (2026). Invoice against purchase order matching by four LLMs, 672 raw decisions.
Think AI First. Test conducted 2026-09-14, published 2026-09-14.
https://think-ai-first.com/datasets/invoice-po-matching-2026-09. Licensed CC BY 4.0.
bibtex
@dataset{thinkaifirst2026pomatching,
  author    = {{Think AI First}},
  title     = {Invoice against purchase order matching by four LLMs, 672 raw decisions},
  year      = {2026},
  publisher = {Think AI First},
  url       = {https://think-ai-first.com/datasets/invoice-po-matching-2026-09},
  note      = {Test conducted 2026-09-14. Licensed CC BY 4.0. Archive SHA-256 559ce436f28746c1baaa4974eef404da61046ed7e9e3ebdd2bb8272b4824c314}
}

Licence and reuse

Published under CC BY 4.0. Use it, quote it, contradict it. If you rerun this and get something different, tell us through the contact form: corrections are published on the article itself, dated.

Download the dataset (zip, 64.5 KiB, 8 files)

Other datasets published here

Each one is a separate test, with its own corpus, its own raw responses and its own archive. This one reuses the invoices of the first and asks a different question of them.