GPT-4o mini in six first-hand tests: 128 of 168 purchase order checks right, 4 of 12 inputs stable at temperature 0

This page gathers the results that GPT-4o mini (openai/gpt-4o-mini) obtained in the tests published on this site between 2026-08-10 and 2026-09-30. It reports no new test. Every figure below was recomputed on 2026-10-19 from the raw responses published with each test, by the script shown at the end of the page, which any reader can run on the same archives.

The tests were conducted by this site's automated test chain: the corpora, the model calls and the analysis are scripts, published with each dataset so that anyone can rerun them and contest the result. In each test, GPT-4o mini received the same instruction and the same inputs as the other models of that test, which is what makes the figures comparable from one model to the next.

The figures, test by test

What each figure describes

Invoice extraction. The model read 24 supplier invoices generated by script, each at four levels of degraded text, and returned seven fields per invoice. 83 of the 96 extractions had all seven fields right. All 13 wrong extractions came back with an empty notes field, although the model was asked to fill it when something in the invoice was unclear. This is the case a human reviewer is least likely to catch.

Self-check. On 2026-09-04 the same model was asked to reread each of its own extractions against the invoice and say whether it was correct. It flagged 13 of its 13 wrong extractions, and it also flagged 26 of its 83 right ones, so a reviewer who opened every flagged invoice would open 39 invoices to find 13 errors.

Rerun. The 96 extractions were run again on 2026-09-06 with a larger output budget, as part of the price against accuracy test: 83 came back fully right, for 0.0095 USD billed. This rerun reproduces the first test and is not counted here as a separate result.

Contact deduplication. Eight address books of 12 contacts, generated by script with planted duplicates and trap pairs (two different people built to look alike), each sent at four levels of formatting mess. Over the 32 passes the model found 87 of the 96 planted duplicates and merged 46 pairs of different people, 43 of them on trap pairs.

Repeatability at temperature 0. The same 12 inputs were sent 10 times each, with the temperature set to 0. For 4 of the 12 inputs the 10 raw answers were identical byte for byte; for one input the model returned 10 different raw answers. The test article also compares the parsed content of the answers, a looser comparison, so its own counts of divergent inputs differ from this byte for byte count.

Support message classification. 270 customer messages taken from the public Bitext support dataset, 10 for each of 27 intents, each to be sorted into one intent with a confidence from 0 to 100. The model named the right intent 237 times. Of its 33 wrong answers, 0 carried a declared confidence of 90 or more.

Invoice against purchase order matching. Each of the 24 invoices was paired with seven purchase orders carrying one controlled discrepancy at a time; for two of the seven conditions, a matching order and a partial delivery, the right decision is to approve. The model decided 128 of the 168 pairs correctly. It approved 3 of the 120 pairs that had to be held, and held 37 of the 48 pairs that had to be approved.

The same inputs, compared with the other two models

Two pages put this model side by side with each of the other two on the inputs all of them received, and count the cases where only one of the two was right:

• Claude Haiku 4.5 vs GPT-4o mini
• Gemini 2.5 Flash Lite vs GPT-4o mini

The pages of the other two models: Claude Haiku 4.5, Gemini 2.5 Flash Lite.

Method

• The test dates are those of the table. This page was published on 2026-10-19, and its figures were recomputed the same day from the archives served on the dataset pages.
• The invoices, the address books, the purchase orders and the 12 repeatability inputs were generated by script with fixed seeds, and each dataset page says so with what that does not prove. The support messages come from the public Bitext dataset.
• The figures of the table come from one call per case and per model, except the repeatability test, which sent each input 10 times.
• Costs are the amounts the API reported for each call, recorded in the raw responses, and not estimates from a price list.

What these figures do not show

• The documents were generated by script and contain no scans, unusual layouts or handwriting, so they do not represent the variety of real documents.
• The results describe the model as it was served on each test date. A provider can update a model under the same name.
• Every case was asked once, and the repeatability test shows that a second call at temperature 0 can return a different answer, so a rerun can move a figure by a few cases.
• The figures describe this model under the instructions of each test. Another instruction could give other results.
• The seven tasks are not weighted against each other, and no overall ranking is drawn from them.

Recompute every figure of the table

This script reads the raw responses of six archives and prints every figure of the table above. Download each archive from the dataset page linked in the table, unzip it into a folder named after the archive (for example unzip 2026-08-invoice-extraction.zip -d 2026-08-invoice-extraction), then run the script from the folder that holds them.

import json
from pathlib import Path

MODEL = "openai/gpt-4o-mini"

def rows(archive, name):
    data = json.loads(next(Path(archive).rglob(name)).read_text())
    rows = data["lignes"] if isinstance(data, dict) else data
    return [r for r in rows if r["modele"] == MODEL]

def cost(rs):
    return round(sum(r["cout_usd"] or 0 for r in rs), 4)

ext = rows("2026-08-invoice-extraction", "resultats-bruts.json")
wrong = [r for r in ext if not r["tous_les_champs_justes"]]
print("extraction: right", len(ext) - len(wrong), "of", len(ext), "| wrong", len(wrong),
      "| wrong with no warning", sum(not r["a_signale_quelque_chose"] for r in wrong), "| USD", cost(ext))

said = {(r["facture"], r["niveau"]): r["dit_correct"]
        for r in rows("2026-09-extraction-guardrails", "verifications.json")}
flag = lambda r: said[(r["facture"], r["niveau"])] is False
print("self-check: wrong flagged", sum(flag(r) for r in wrong),
      "| right flagged", sum(flag(r) for r in ext if r["tous_les_champs_justes"]))

rerun = rows("2026-09-invoice-price-accuracy", "passeB-fusionnee.json")
print("rerun: right", sum(r["tous_les_champs_justes"] for r in rerun), "of", len(rerun), "| USD", cost(rerun))

dd = rows("2026-08-contact-deduplication", "resultats-bruts.json")
print("deduplication: passes", len(dd), "| found", sum(r["nb_fusions_justes"] for r in dd),
      "of", sum(r["nb_fusions_justes"] + r["nb_fusions_manquees"] for r in dd),
      "| false merges", sum(r["nb_fusions_fausses"] for r in dd),
      "| traps", sum(r["nb_pieges_tombes"] for r in dd))

raw = {}
for r in rows("2026-08-temperature-zero-repeatability", "resultats-bruts.json"):
    raw.setdefault(r["entree"], set()).add(r["brut"])
print("temperature 0: identical", sum(len(v) == 1 for v in raw.values()), "of", len(raw),
      "| most variants", max(len(v) for v in raw.values()))

tk = rows("2026-09-support-ticket-classification", "resultats-bruts.json")
print("tickets: right", sum(bool(r["juste"]) for r in tk), "of", len(tk),
      "| wrong at confidence 90+", sum(not r["juste"] and (r["confiance"] or 0) >= 90 for r in tk),
      "| USD per 1,000", round(1000 * sum(r["cout_usd"] or 0 for r in tk) / len(tk), 4))

po = rows("2026-09-invoice-po-matching", "resultats-bruts.json")
print("purchase orders: right", sum(bool(r["juste"]) for r in po), "of", len(po),
      "| wrong APPROVE", sum(not r["juste"] and r["decision"] == "APPROVE" for r in po),
      "| wrong HOLD", sum(not r["juste"] and r["decision"] == "HOLD" for r in po))