THINK AI FIRST

Dataset

Two error detection guardrails on 368 proposed contact merges

96 deduplication runs already published here, five guardrails applied to every merge they proposed, and the 96 raw responses of the one that needed model calls. Test conducted on 2026-09-12.

This page describes one dataset, published in full. It is the raw material behind our test of two guardrails on a task with no arithmetic: 96 deduplication runs already published here, five guardrails applied to every merge they proposed, and the 96 raw responses of the guardrail that needed model calls.

Everything on this page comes from that one run. Nothing is estimated.

What it measures, and what it does not prove

It measures no model. It measures the checks you would put around one to catch a false merge before two people become one row.

The question it was built for is narrow and it is the whole point: on 2026-09-04 we measured four guardrails on invoice extractions, where a free arithmetic identity exists. A contact list has none. This dataset re-runs the two guardrails that held up there, on a task where nothing can be recomputed, and scores them the same way.

Each guardrail carries two quantities that are never read one without the other: the catch rate, the share of the wrong merges it flags, and the false alarm rate, the share of the correct merges it flags. A guardrail that flags everything scores 100 % on the first and is useless. One that flags nothing scores 0 % on the second and is useless too.

What it does not prove, stated plainly, because a dataset quoted without its limits does more harm than no dataset at all:

  • The 0 % false alarm rate of the key rule is true by construction. The source corpus requires every duplicate pair to share an email or a phone. In a real address book, somebody who changed both would have their merge blocked. This is the most flattering number in the file and it is the one to distrust first.
  • The partitions are 26 days old. They were produced on 2026-08-17. Nothing here says the same models make the same merges today.
  • 97 wrong merges. Every catch rate rests on that. A percentage on 97 items has a wide margin: orders of magnitude, not constants.
  • 2 of the 96 responses are not valid JSON, and 7 of the 366 submitted groups came back with no verdict. Those 7 merges leave the self-check population, which is scored on 96 wrong and 265 correct instead of 97 and 271. Declared, not repaired.
  • One corpus, 96 constructed rows, fixed seed, declared as constructed in its own protocol. No real person, no real company. The trap families are designed features of that corpus and their frequency here is not their frequency in a CRM.
  • Address books of 12 rows, entirely inside the model window: the most favourable case. A 5000 row list is a different problem, not measured here.
  • The self-check prompt is one prompt. A different instruction would give different numbers. The one that was run is published inside verifications.json, rather than a flattering variant found afterwards.
  • The comparison with the invoice test puts two tasks and two corpora side by side. What changes is not only the presence of an arithmetic check, and this dataset cannot separate the two.

The partitions are reused, and here is what that means

Nothing was deduplicated for this test. The 96 partitions judged here are exactly those published on 2026-08-18 under the contact deduplication dataset: 8 script generated address books of 12 rows from seed 20260817, four levels of formatting mess, three low cost models, and a ground truth partition settled then.

That is a deliberate trade and it cuts both ways. What it buys: all five guardrails judge an identical set of merges, so the comparison isolates the guardrails and nothing else. What it costs: rerunning the models on 2026-09-12 would have mixed their effect with 26 days of model drift, and this dataset says nothing about the models as they are today.

This dataset generates no corpus of its own. Its reproducibility rests on the seed of the source corpus, 20260817. Regenerate it and you hold the same 8 address books.

Test conducted on 2026-09-12. The calls were run by the automated testing pipeline of this site: the self-check pass and the analysis are scripts, and both are in the archive below.

The unit that is scored

Not an address book, not a call: a proposed merge, meaning a pair of rows a model put in the same group.

Count
Merges proposed across the 96 runs 368
Correct, two rows and one person 271
Wrong, two people and one row destroyed 97

The guardrails

What it does Cost
A0 the model's own notes field, filled while deduplicating: does it name both rows? none
A1 strong keys: refuse any merge whose rows share neither email nor phone none
A3 self-check: one extra call to the same model, given the list and its own groups one call per run
A4 agreement of three models: flag any merge the other two did not both propose two extra runs
A5 agreement with a single other model, computed for all three witnesses one extra run

The A3 instruction never hints that an error was found and gives no ground truth, and the verdict is asked group by group rather than on the whole partition.

Raw figures

368 merges judged, 97 wrong, 271 correct.

Guardrail Catch rate False alarms Precision Wrong merges missed
A0 spontaneous note 73.2 % 55.4 % 32.1 % 26
A1 strong keys 72.2 % 0.0 % 100.0 % 27
A3 self-check 76.0 % 20.8 % 57.0 % 23
A4 agreement of three models 62.9 % 9.2 % 70.9 % 36
A5 with gpt-4o-mini alone 25.5 % 8.7 % 44.8 % 38
A5 with gemini-2.5-flash-lite alone 40.0 % 3.9 % 78.8 % 39
A5 with claude-haiku-4.5 alone 65.4 % 2.8 % 91.1 % 27
A1 and A4 together 90.7 % 9.2 % 77.9 % 9
A1 and A3 together 92.7 % 20.8 % 61.8 % 7
A3 and A4 together 87.6 % 27.4 % 53.8 % 12

The same guardrails on the invoice task

Guardrail Invoices, 2026-09-04 Contacts, 2026-09-12
Self-check 97.6 % caught / 39.0 % false 76.0 % caught / 20.8 % false
Agreement of three models 92.9 % caught / 11.0 % false 62.9 % caught / 9.2 % false
The free mechanical rule 59.5 % caught / 22.0 % false 72.2 % caught / 0.0 % false

By model, on the agreement guardrail

Model Wrong merges proposed A4 catch rate A4 precision
openai/gpt-4o-mini 46 of 133 73.9 % 87.2 %
google/gemini-2.5-flash-lite 32 of 124 62.5 % 66.7 %
anthropic/claude-haiku-4.5 19 of 111 36.8 % 41.2 %

By level of formatting mess

Level Merges proposed Wrong A1 caught A3 caught A4 caught A4 false alarms
N1 clean 82 18 66.7 % 88.2 % 83.3 % 10.9 %
N2 case and spaces 95 26 69.2 % 65.4 % 53.8 % 8.7 %
N3 phone formats mixed 104 33 72.7 % 78.8 % 63.6 % 2.8 %
N4 columns merged 87 20 80.0 % 75.0 % 55.0 % 14.9 %

Reading load per 1000 proposed merges

In a population where 264 of them are wrong, which is this corpus and not a real address book.

Guardrail To re-read Wrong merges found Missed Reads per one found
A1 190 190 73 1.0
A3 355 202 64 1.8
A4 234 166 98 1.4
A1 and A4 307 239 25 1.3

Costs

The self-check pass, 96 calls: 0.07469 USD, 96 read, 94 valid JSON, 359 verdicts returned for 366 submitted groups. Per 1000 self-check calls: gpt-4o-mini 0.196 USD, gemini-2.5-flash-lite 0.161 USD, claude-haiku-4.5 1.977 USD.

Every verdict was matched to its group by equality of the row sets, 359 times out of 359, and never by position. A positional match would have been an assumption; it is counted separately in chiffres.json and it is zero.

What the archive contains

Eight files, 492862 bytes uncompressed. The inventory is here so you know what you are downloading before you download it.

File What it holds Size
corpus.json the 8 address books, the ground truth and the explanation of each pair, seed 20260817 40.6 KiB
degrader.py the four formatting levels, needed to hand the model the exact list it saw 3.6 KiB
resultats-bruts.json the 96 partitions being judged, published 2026-08-18 213.3 KiB
auto-verifier.py the self-check pass, one call per partition 7.5 KiB
verifications.json the 96 raw self-check responses, with the instruction used 172.8 KiB
analyser.py the five guardrails, their combinations and the reading load 15.6 KiB
chiffres.json every computed figure 14.4 KiB
PROTOCOLE.md the protocol, the reading of the figures, and the limits 13.5 KiB

corpus.json, degrader.py and resultats-bruts.json are copied unchanged from the 2026-08-18 dataset, so this archive stands on its own without fetching another one.

The scripts and their comments are in French, which is the working language behind this site. The API key is read from an environment variable and appears in none of these files.

Rerun it

The analysis calls no model at all, so it replays at zero cost on the raw responses shipped in the archive. Only auto-verifier.py spends anything.

bash
unzip 2026-09-deduplication-guardrails.zip
python3 auto-verifier.py --resultats resultats-bruts.json --corpus corpus.json \
    --sortie verifications.json
python3 analyser.py --resultats resultats-bruts.json \
    --verifications verifications.json --sortie chiffres.json

A guardrail rule you disagree with can be changed in analyser.py and rerun on the same 96 responses without paying for a single call.

Each raw response is one entry of verifications.json, exactly as the model returned it, before any scoring. This one holds both failure directions in a single answer: the merge of two different people is confirmed, and the merge of two rows sharing one email address is rejected.

json
{"modele": "openai/gpt-4o-mini", "carnet": "carnet-01", "niveau": 1,
 "lu": true, "json_valide": true,
 "groupes_soumis": [[4, 5], [3, 8], [6, 9], [1, 10]],
 "verdicts": [{"rows": [4, 5], "same_person": true},
              {"rows": [3, 8], "same_person": true},
              {"rows": [6, 9], "same_person": false},
              {"rows": [1, 10], "same_person": false}]}

Verify what you downloaded before you trust it:

bash
shasum -a 256 2026-09-deduplication-guardrails.zip
# 245aa5fb0be4b00f99793ec677118123d56c9a8669695f4e1d4f98cf0c9c407b

How to cite this dataset

Copy one of the two forms below. The reference points at this page, which is the stable address of this dataset; the archive is what that address serves.

citation
Think AI First (2026). Two error detection guardrails on 368 proposed contact merges, 96 raw self-check responses.
Think AI First. Test conducted 2026-09-12, published 2026-09-12.
https://think-ai-first.com/datasets/deduplication-guardrails-2026-09. Licensed CC BY 4.0.
bibtex
@dataset{thinkaifirst2026dedupguardrails,
  author    = {{Think AI First}},
  title     = {Two error detection guardrails on 368 proposed contact merges, 96 raw self-check responses},
  year      = {2026},
  publisher = {Think AI First},
  url       = {https://think-ai-first.com/datasets/deduplication-guardrails-2026-09},
  note      = {Test conducted 2026-09-12. Licensed CC BY 4.0. Archive SHA-256 245aa5fb0be4b00f99793ec677118123d56c9a8669695f4e1d4f98cf0c9c407b}
}

Licence and reuse

Published under CC BY 4.0. Use it, quote it, contradict it. If you rerun this and get something different, tell us through the contact form: corrections are published on the article itself, dated.

Download the dataset (zip, 48.3 KiB, 8 files)

Other datasets published here

Each one is a separate test, with its own corpus, its own raw responses and its own archive. This one judges the merges of the second.