Claude Haiku 4.5 vs Gemini 2.5 Flash Lite on the same inputs: where each one is right and the other is wrong, in three first-hand tests

This page puts Claude Haiku 4.5 and Gemini 2.5 Flash Lite side by side on the inputs that both models received in three tests published on this site, and counts, case by case, where one of the two was right and the other wrong. It reports no new test. The figures were recomputed on 2026-10-19 from the raw responses published with each test, by the script shown at the end of the page.

The tests were conducted by this site's automated test chain: the corpora, the model calls and the analysis are scripts, published with each dataset so that anyone can rerun them and contest the result. Within each test, both models received the same instruction and the same inputs.

Case by case, in three tests

Reading the table

Invoice extraction. Out of 96 extractions that both models handled, Claude Haiku 4.5 was right on 1 where Gemini 2.5 Flash Lite was wrong, and Gemini 2.5 Flash Lite was right on 8 where Claude Haiku 4.5 was wrong. Both models were wrong on 10.

Support message classification. Out of 270 messages that both models handled, Claude Haiku 4.5 was right on 12 where Gemini 2.5 Flash Lite was wrong, and Gemini 2.5 Flash Lite was right on 27 where Claude Haiku 4.5 was wrong. Both models were wrong on 14.

Invoice against purchase order matching. Out of 168 pairs that both models handled, Claude Haiku 4.5 was right on 10 where Gemini 2.5 Flash Lite was wrong, and Gemini 2.5 Flash Lite was right on 3 where Claude Haiku 4.5 was wrong. Both models were wrong on 24.

When only one of the two models is right, their two answers necessarily differ, so a check that compares both answers would at least notice those cases. When both are wrong, the two answers can agree, and such a check may notice nothing. The last column therefore bounds what asking the second model as well could recover on these inputs.

The full results of each model, including the tests that are not compared here, are on its own page: Claude Haiku 4.5 and Gemini 2.5 Flash Lite.

Method

• The test dates are those of the table. This page was published on 2026-10-19, and its figures were recomputed the same day from the archives served on the dataset pages.
• A case counts as right under the rule of its own test: all seven fields right for an extraction, the expected intent for a support message, the expected decision for a purchase order pair.
• The invoices and the purchase orders were generated by script with fixed seeds; the support messages come from the public Bitext dataset.
• The rerun of the invoice extraction of 2026-09-06 gives the same counts as the first run for this pair and is not added as a fourth test.

What these figures do not show

• Each case was asked once to each model. A second call at temperature 0 can return a different answer, as this site measured on 2026-08-25, so a rerun can move each count by a few cases.
• The invoices and purchase orders were generated by script and do not represent the variety of real documents.
• The results describe both models as they were served on each test date and under the instructions of each test.

Recompute the table

This script reads the raw responses of three archives and prints the counts of the table, except the both right column, which is the cases in common minus the three other columns. Download each archive from the dataset page linked in the table, unzip it into a folder named after the archive, then run the script from the folder that holds them.

import json
from pathlib import Path

A, B = "anthropic/claude-haiku-4.5", "google/gemini-2.5-flash-lite"
TESTS = [
    ("2026-08-invoice-extraction", lambda r: (r["facture"], r["niveau"]), lambda r: r["tous_les_champs_justes"]),
    ("2026-09-support-ticket-classification", lambda r: r["message"], lambda r: bool(r["juste"])),
    ("2026-09-invoice-po-matching", lambda r: r["paire"], lambda r: bool(r["juste"])),
]

for archive, key, right in TESTS:
    data = json.loads(next(Path(archive).rglob("resultats-bruts.json")).read_text())
    rows = data["lignes"] if isinstance(data, dict) else data
    a = {key(r): right(r) for r in rows if r["modele"] == A}
    b = {key(r): right(r) for r in rows if r["modele"] == B}
    common = sorted(set(a) & set(b))
    print(archive, "| common", len(common),
          "| only A right", sum(a[k] and not b[k] for k in common),
          "| only B right", sum(b[k] and not a[k] for k in common),
          "| both wrong", sum(not a[k] and not b[k] for k in common))