Temperature 0 is not determinism: 360 calls, 12 silent divergences

Temperature 0 is the setting everybody reaches for when a model has to sit inside a workflow. Set it to zero, the sampling stops, the output becomes deterministic, and the step can be trusted. That is the reasoning, and it is why a lot of automations go to production after one successful run.

We sent the same input to the same model ten times, at temperature 0, with the instruction identical to the character. 12 inputs, 3 models, 360 calls. Here is what came back, before the method.

12 cells out of 36 gave more than one answer. A cell is one input paired with one model, so ten calls that should have been ten copies of each other. A third of them were not.

All 12 divergences were silent. Across 360 calls there was not one unreadable JSON, not one missing key, not one wrong type, not one truncation. A schema check placed at the output of the step would have passed 12 divergences out of 12. Nothing in an ordinary workflow would have raised a hand.

The output shape decides more than the model does. Closed field extraction, where the number of fields is known in advance, was perfectly repeatable: 120 calls, one answer. Free length output was stable in 5 cells out of 12.

One model out of three was stable everywhere, and that breaks the headline. anthropic/claude-haiku-4.5 returned the same payload on all 12 of its cells, including the open shape where the other two collapsed. "Temperature 0 is not determinism" is true for two of these three models and false for the third, on this perimeter. Anyone stating the rule without naming the model is wrong half the time.

Method, in short

Test conducted on 2026-08-25 by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, published below so that anyone can rerun them and disagree with the result. Etienne signs this article and answers for its content.

This is the fourth measured test published here, and the first that does not measure correctness. The first measured field extraction, the second a merge decision, the third an open list. All three asked whether the model was right. This one asks whether it says the same thing twice.

The same input goes to the same model 10 times, temperature 0, max_tokens 1600, through OpenRouter. Nothing distinguishes the ten calls: no sequence number, no timestamp, no uniqueness token. That is the point of the test. Adding anything that varies would measure sensitivity to input noise instead.

The three models are the ones used in the previous tests of this site: openai/gpt-4o-mini, google/gemini-2.5-flash-lite, anthropic/claude-haiku-4.5. Total measured cost: 0.247 USD.

The inputs are not new, and that is deliberate

No data was created for this test. The 12 inputs are taken from the three corpora already published on this site, at degradation level 1, the clean level, the most favourable of the four. If repeatability already gives way on the easiest case, it does not improve on the harder ones.

Those three corpora give three output shapes, and the shape is the variable that matters here.

Four items per shape, the first four of each corpus, no draw. The selection is deterministic and replays identically. The instructions are copied word for word from the original tests: changing one would measure something other than what those tests measured.

The corpora are generated from a fixed seed and declared as constructed in their own protocols. No real invoice, no real address book, no real meeting.

How a divergence is classified

The calling script classifies nothing. It records the raw response. All classification is done afterwards by analyser.py from the raw file, so every figure recomputes without calling a model again, and a rule you disagree with can be changed and replayed on the same 360 responses.

Three comparison levels, and all three numbers are given because the choice between them is arguable and it changes the result:

• whole: the complete response, notes field included. The strictest.
• payload: the data fields alone, without notes. This is what a workflow actually consumes.
• semantic: the payload after normalisation. Numbers rounded to two decimals, duplicate groups read as a set of sets, actions sorted by their text. Two responses differing only by order count as identical here. The most forgiving.

Then, on the cells that diverge:

• visible: at least one of the 10 responses fails structural validation. Ordinary error handling sees it and can retry.
• silent: the cell diverges and all 10 responses pass structural validation. Nothing in the workflow sees it.

Raw figures

By comparison level

9 cells out of 36 diverge even after semantic normalisation. Those are not ordering differences, they are different answers. The 3 remaining differ only by order, and still count as divergent at payload level because a consumer comparing strings would see them as different.

By output shape

By model

Every cell that diverged

Twelve rows, the whole list, nothing selected. variants counts distinct payloads across the 10 calls. majority is the share held by the most frequent one. valid is how many of the 10 passed structural validation.

The Valid column is the finding. It reads 10/10 on every row.

Five different answers in ten identical calls

openai/gpt-4o-mini on binaire-carnet-04, the address book deduplication task. Ten identical calls, five distinct groupings. Rows 8 and 9 are merged in 5 calls out of 10 and left separate in the other 5. The group around rows 3, 4 and 10 changes composition from one call to the next.

rank  1  [["1","7"],["2","12"],["3","4","10"],["5","6"],["8","9"]]
rank  2  [["1","7"],["2","12"],["3","4"],["6","5"],["8","9"]]
rank  3  [["1","7"],["2","12"],["4","10"],["6","5"]]
rank  4  [["1","7"],["2","12"],["3","4"],["6","5"],["8","9"]]
rank  5  [["1","7"],["2","12"],["4","10"],["6","5"]]
rank  6  [["1","7"],["2","12"],["3","4"],["6","5"],["8","9"]]
rank  7  [["1","7"],["2","12"],["3","4","10"],["6","5"]]
rank  8  [["1","7"],["2","12"],["3","4","10"],["6","5"],["8","9"]]
rank  9  [["1","7"],["2","12"],["4","10"],["6","5"]]
rank 10  [["1","7"],["2","12"],["3","4","10"],["6","5"],["8","9"]]

Translated into consequence: the cleanup you ran on Monday merges two records, the one you run on Tuesday does not, and the JSON is valid on both days.

Nine times the same answer, then something else

google/gemini-2.5-flash-lite on binaire-carnet-01. Ranks 1 to 9 return exactly the same response. Rank 10 adds two groups, one of them ["4","5"], which the ground truth of the original corpus marks as a trap: two exact homonyms, same first name, same last name, two companies, two cities, two email addresses, two phone numbers. Two different people, merged on the tenth call and not on the previous nine.

The raw file makes it worse than a coin flip, and better documented. On rank 9, the model wrote this in its own notes field, about the very pair it would merge one call later:

Row 5 and 4 have the same first name and last name, but different
companies and emails, suggesting they are different people.

On rank 10 it put rows 4 and 5 in the same group. Same input, same instruction, same temperature. The model stated the correct reasoning nine times and acted against it on the tenth, and both responses are valid JSON with a filled notes field. A reviewer reading the notes of the tenth call would find a confident explanation of the merge, not a warning.

That is the one in ten event a single test does not see, and it merges two distinct human records, which is not easy to undo in a CRM.

What "I tested it once and it worked" is worth

On the 24 cells whose ground truth is cheap and uncontroversial, closed and binary, the value of a single trial can be computed exactly.

Read the last line this way: pick one call at random among all the calls that came out right, and ask whether its cell is right on all 10. The answer is yes with probability 0.947 overall and 0.816 on the decision task. On that task, roughly one green trial in five covers a step that is not reliable.

Repeatable and wrong are not opposites

The other half of that table is harder. On the binary task, 7 cells out of 12 are wrong on all 10 calls. The model gets them wrong with perfect stability.

So repeatability is not correctness, and a step can be repeatable and systematically wrong. The test of 2026-08-18 already measured that; it is found again here by another route. Ten identical answers tell you the step is stable. They tell you nothing about whether it is right.

What we would do with this

Three things follow from the figures above, and nothing else does.

Close the output shape wherever the task allows it. The closed shape is the only one that came back perfectly repeatable, on all three models, 120 calls for one answer. A fixed list of fields, and a task that fills them, is worth more here than a free form answer that is later parsed.

Validate values, not just structure. Every single divergence in this test passed a structural check. A validation that verifies bounds, totals that must add up, identifiers that must already exist, would have seen part of what the schema check missed. That is the practical conclusion, and it is also a limit of this test: our classification of silent versus visible depends on the validation we assume.

Run the step more than once before trusting it, on the decision tasks. Not in production, once, in evaluation. Ten runs of one input cost fractions of a cent here and moved the reliability estimate on the binary task from "it worked" to 0.816.

Limits, stated plainly

This section is not a formality. It is the condition for the rest to be readable.

• One day, one hour, one provider. The 360 calls went through OpenRouter on 2026-08-25. Nothing here says what the same models return in a month, or what a direct call to the provider returns. The repeatability measured is intra day.
• 12 inputs. Four per shape. A 41.7 % rate computed on 12 cells has a wide margin. It is an order of magnitude, not a constant.
• claude-haiku-4.5 at 12/12 does not mean deterministic. It means that on these 12 inputs, that day, its 120 calls landed on the same payload. Twelve cells do not demonstrate a property. We checked it is not a cache: on binaire-carnet-03 its ten responses have an identical payload and a different notes field. A cache would have returned the whole response identical. The model resampled ten times and landed in the same place.
• Constructed corpora. Fixed seed, declared as constructed in their own protocols.
• Nothing about long contexts. The three corpora fit comfortably in the context window of all three models. The favourable case again.
• The open shape is not scored for correctness. Its ground truth needs the scoring rules of the 2026-08-18 test, which are arguable and are not the subject here. Only its repeatability is reported.

Rerun it

Everything this test produced is published, not summarised: the builder that assembles the 12 inputs from the three corpora with their instructions, the calling script, the 360 raw responses exactly as the models returned them, and the analysis that produces every figure above.

python3 conduire-test.py --sortie resultats-bruts.json --repetitions 10 \
    --modeles openai/gpt-4o-mini google/gemini-2.5-flash-lite anthropic/claude-haiku-4.5
python3 analyser.py --brut resultats-bruts.json --sortie chiffres.json

The analysis alone calls no model, so it replays at zero cost. --rejouer resultats-bruts.json only recalls the lines marked lu: false. The API key is read from an environment variable and appears nowhere in these files.

Download the dataset (zip, with the full inventory and the checksum). The scripts and the comments inside them are in French, which is the working language behind this site.

If you rerun this and get something different, we want to know. There is a contact form, and corrections get published on the article itself, dated, as the editorial rules of this site say.