LLM contact deduplication: 96 tests on how often it merges two different people

A small business exports its contact list and would rather have a model clean it up than scroll through it. Whether that works is not really the question. It works, and we measured that too. The question is what happens when the model decides two different people are one person, and whether anybody finds out.

We ran 96 deduplication passes to find out. Here are the results, before the method.

Three cheap models found 271 of 288 real duplicates, 94.1 percent. claude-haiku-4.5 found all 24 on the three degraded versions of the file. Finding duplicates is not the hard part, and nothing in this test suggests it is.

The same models merged two different people in 93 of 192 chances, 48.4 percent. Nearly one look-alike pair in two was fused. A missed duplicate leaves two rows in a file and costs nothing. A false merge destroys a row: one of the two people loses their address, their job title and their history, and the file that comes back does not say so.

Of the 97 false merges, 68 were silent, 70.1 percent. Silent here does not mean the model said nothing. It means that nothing in what it said would send an operator to look at that particular pair. In 42 of those 68 cases the model named the two rows it had just wrongly fused and stated, with no hedge of any kind, that they were the same person.

The clean file was not the safest one. False merges rose from 33.3 percent on the pristine CSV to 66.7 percent on the version where only the phone number formatting varied. Correct merges rose at the same time. The messier the file, the more the model merges, right and wrong together.

Method, in short

Test conducted on 2026-08-17 by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, and they are published below so that anyone can rerun them and disagree with the result. Etienne signs this article and answers for its content.

This is the second measured test published here. The first one measured field extraction from a document. This one measures a decision. Merging two rows is not reading a value: it is settling an identity, and the mistake is not recoverable the same way.

The corpus is built, not collected. 8 address books of 12 rows, 96 contacts, generated by script with the fixed seed 20260817, so the corpus regenerates identically. No real person, no real company, no real address. Phone numbers come from the 555-01xx range reserved for fiction and email domains sit under the reserved .example TLD.

That is a deliberate trade. The ground truth of a deduplication is a partition, and it is only beyond argument if we are the ones who set it. On a real address book we would have had to settle hundreds of pairs by hand, with our own judgement errors inside the reference, and the test would have measured our agreement with the model rather than the model's correctness.

Each address book holds 3 duplicate pairs to merge, 2 look-alike pairs to leave alone, and 2 single rows. Across the 8 books: 24 duplicate pairs and 16 look-alike pairs.

Every duplicate pair shares at least one strong key, the email or the phone, so that a single defensible answer exists. Every look-alike pair differs on at least two attributes derivable from the book itself. A check script verifies both invariants and the absence of accidental collisions before a single call is made.

Three models, picked among those a small business would actually use for their price: openai/gpt-4o-mini, google/gemini-2.5-flash-lite, anthropic/claude-haiku-4.5. Access through OpenRouter, temperature 0, one run per combination of book, level and model. 96 calls, 0 technical failures, 0 invalid JSON responses, total API cost 0.06647 USD. The instruction was identical word for word across the three models, it offered a notes field for flagging a hesitation, and it stated explicitly that merging two different people is worse than leaving a duplicate in the list.

What this does not prove. Our books hold 12 rows and fit entirely inside the model's context, which is the most favourable case; a 5000 row file is a different problem this test does not touch. English only. Latin names only. One run per combination. This is a test on 96 synthetic rows, not a production audit. The full list of limits is at the end and it is worth reading before quoting any figure here.

What we asked the models to trip over

Five families of duplicates, each anchored by a key that makes the right answer defensible.

Three families of look-alikes, each of them two genuinely different people.

T2 is there for a precise reason. The phone number anchors three of the five duplicate families, and T2 checks whether the model treats it as an identity. A switchboard number is shared by a whole office.

An objection we take seriously, and price. Two exact namesakes at two different companies could be one person who changed employer, and the address book does not settle it to the last percent. So here is the figure without that family: 50 wrong merges out of 120, 41.7 percent. The conclusion holds even if you give the reader that point.

The four levels of mess

An export is never clean. We degrade the presentation, never the information: at every level the same letters and the same matches remain derivable. An export that genuinely lost an email would no longer measure the model's judgement, it would measure what we hid from it.

The degradation is deterministic. The same book produces the same string on every run.

Raw results

What works

The spread between models is real here: claude-haiku-4.5 makes 19 false merges where gpt-4o-mini makes 46, while both sit within five points of each other on duplicates found. On this task the choice of model changes the risk by more than a factor of two and barely changes the yield.

Of the 97 false merges, 93 landed on the 16 look-alike pairs we designed. The other 4 were pairs we had not planned at all: rows the model brought together for reasons of its own. We report them because they are in the data, and we draw nothing from four cases.

The clean file is not the safest

Both columns climb together from N1 to N3. The model does not get worse as the file gets dirtier: it gets more aggressive. It stops leaning on identity keys and starts leaning on resemblance, exactly when resemblance is least reliable. The advice to tidy the file before handing it over is not wrong, but it is worth knowing what it actually buys: on this corpus, cleaning up cut the false merges roughly in half and cost seven correct merges.

The one number that surprised us is N3 against N4. Rewriting the same phone number four different ways did more damage than collapsing every column into one string. Formatting variation inside a field appears to be harder for these models than the loss of the field structure itself.

Silence has four degrees

A note is not a warning. At the very first trial call, a model merged two exact namesakes and wrote a note asserting they were the same person. Counting that as a flag would have made the measurement wrong in the flattering direction. So we classify each false merge by what its note actually gives an operator:

Degrees 1, 2 and 3 are what we call silent: 68 of 97, 70.1 percent. In none of the three does anything call for a check on that specific pair.

The doubt lexicon is published in full in the analysis script. It is deliberately generous: it counts "appears" or "likely" as a doubt. Being generous works against our own conclusion, and that is the only acceptable direction for a lexicon chosen by whoever publishes the number.

Two models never left the field empty at all: gpt-4o-mini and claude-haiku-4.5 wrote a note on all 32 of their responses. The empty notes are all gemini-2.5-flash-lite, 12 of its 32. Writing a note is not the same thing as raising a doubt, and this is the row of numbers that shows it.

The rule that catches two thirds of it

Stated in one line: refuse any merge whose two rows share neither the same email, case ignored, nor the same phone digits.

That zero is true by construction, and it has to be said. Our corpus requires every duplicate pair to share an email or a phone. In a real address book, a person who changed both their address and their number exists, and this rule would block that merge. The honest statement is therefore: the rule catches 72.2 percent of the false merges at the cost of a recall you have to measure on your own file, not at the cost of nothing.

The 27 it lets through are the T2 family, the ones sharing a switchboard number. For those there is a second rule, just as cheap: if two rows share a phone but differ on the first name, they are two people until proven otherwise.

What to do with this

Do not let a model merge. Let it propose. The yield is on its side, 94.1 percent, and the risk is on yours, one in two look-alike pairs. Take the proposal, run the strong key check on it, and merge only what passes.

Do not read the notes as warnings. In 42 cases out of 97, the note pointed at the exact pair and confirmed the mistake in a confident tone. A note is a comment, not a control.

Keep the pre-merge file. This is the one recommendation that costs nothing and covers everything the two rules miss. A false merge is only irreversible if you no longer have the row it ate.

If you want the broader version of this reasoning, we wrote about the tasks where AI fails without saying so and about the failure modes of automated workflows. This test is one measured instance of both.

Limits, stated plainly

• 96 synthetic rows in 8 books. A test on 8 books is a test on 8 books.
• Books of 12 rows, entirely inside the model's context. A 5000 row file requires chunking, blocking and a comparison strategy, none of which is measured here, and all of which can only make the problem harder.
• One run per combination, at temperature 0. Models are not perfectly deterministic even so. Gaps of a few points should not be read as significant. The gap between 19 and 46 false merges, and above all the shape of the errors, always the same trap families, are.
• Three entry level models. Nothing here says what the more expensive ones would do. That is an open question and it deserves its own test.
• English only, Latin names only. Transliterated names and genuinely homonymous companies are a known hard case this corpus does not contain.
• The doubt lexicon is a choice. It is published, it is generous, it remains a choice.
• The corpus is ours. It is published so that disagreement is possible.

Rerun it

Everything this test produced is here, not summarised: the script that generated the 8 address books and the one that degraded them, the ground truth with an explanation of every pair, the 96 raw responses exactly as the models returned them, and the analysis that produces every figure above.

Seven files: the corpus generator with its fixed seed, the 8 books with their ground truth and the reason each pair is what it is, the four presentation levels, the script that called the models and compared partition against partition, the 96 raw responses with the prompt used, and the analysis that classifies the silence. The data page lists them one by one with what each holds.

The generator uses a fixed seed, so the corpus you produce is the corpus we used.

python3 generer-corpus.py --graine 20260817 --sortie corpus.json
python3 conduire-test.py --corpus corpus.json --sortie resultats-bruts.json \
    --niveaux 1 2 3 4 \
    --modeles openai/gpt-4o-mini google/gemini-2.5-flash-lite anthropic/claude-haiku-4.5
python3 analyser.py --corpus corpus.json --resultats resultats-bruts.json \
    --sortie chiffres.json

The API key is read from an environment variable and appears nowhere in these files.

Download the dataset (zip, with the full inventory and the checksum). The scripts and the comments inside them are in French, which is the working language behind this site.

If you rerun this and get something different, we want to know. There is a contact form, and corrections get published on the article itself, dated, as the editorial rules of this site say.