THINK AI FIRST

Dataset

Repeatability of an LLM step at temperature 0

360 raw model responses, 12 inputs reused from the three corpora already published here, 3 models, 10 repetitions each. Test conducted on 2026-08-25.

This page describes one dataset, published in full. It is the raw material behind our measurement of what temperature 0 actually guarantees: 12 inputs, 3 models, 10 repetitions of every pair, and the 360 responses those models actually returned.

Everything on this page comes from that one run. Nothing is estimated.

What it measures, and what it does not prove

It measures one thing precisely: whether the same input, sent to the same model ten times at temperature 0 with an identical instruction, comes back as the same answer. And then, on the pairs where it does not, whether anything in the output would tell a workflow that something moved.

It is the first dataset published here that does not measure correctness. The three before it asked whether the model was right. This one asks whether it is repeatable, which is a different property and, as the figures show, an independent one.

What it does not prove, stated plainly, because a dataset quoted without its limits does more harm than no dataset at all:

  • One day, one hour, one provider. The 360 calls went through OpenRouter on 2026-08-25. Nothing here says what the same models return in a month.
  • 12 inputs, four per output shape. A rate computed on 12 cells has a wide margin. It is an order of magnitude, not a constant.
  • Level 1 inputs only, the clean level of the three source corpora, the most favourable of the four available.
  • The inputs come from corpora built by script, with fixed seeds, declared as constructed in their own protocols. No real invoice, no real address book, no real meeting.
  • Nothing about long contexts. All three corpora fit comfortably inside the context window of all three models.
  • The classification of silent versus visible depends on the validation we assume. Ours checks keys, types, and for the binary task that each group holds at least two identifiers. A richer validation would see more.

The inputs are reused, and here is what that means

Nothing was created for this test. The 12 inputs are the first four items of each of the three corpora already published on this site, taken at degradation level 1, without a draw. The selection is deterministic and replays identically. The instructions are copied word for word from the original tests: changing one would measure something other than what those tests measured.

That is a deliberate trade and it cuts both ways. What it buys: the ground truth already exists, already published, already contested if anyone wishes to. What it costs: the perimeter is exactly the perimeter of the earlier tests, and inherits their limits along with their data.

This dataset has no seed of its own, because it generates nothing. Its reproducibility rests on the seeds of the three source corpora, which are 20260810 for the invoices, 20260817 for the address books and 20260818 for the meetings. Regenerate those three, take the first four items of each at degradation level 1, and you hold the exact 12 inputs used here.

The three source corpora give three output shapes, which is the variable under test: closed (7 invoice fields, size known in advance), binary (group the duplicates in an address book, a decision), and open (list the action items in meeting notes, free length).

Test conducted on 2026-08-25. The calls were run by the automated testing pipeline of this site: the input builder, the model calls and the analysis are scripts, and all of them are in the archive below.

Raw figures

Stability by comparison level

Comparison level Stable cells out of 36 Share
whole, notes included 19 52.8 %
payload, notes excluded 24 66.7 %
semantic, order ignored 27 75.0 %

By output shape, payload level

Shape Stable cells Mean variants Max variants Silent divergences
closed 12 / 12 1.00 1 0
binary 7 / 12 1.75 5 5
open 5 / 12 2.08 6 7

By model, payload level

Model closed binary open overall Silent divergences
anthropic/claude-haiku-4.5 4/4 4/4 4/4 12/12 0
openai/gpt-4o-mini 4/4 2/4 1/4 7/12 5
google/gemini-2.5-flash-lite 4/4 1/4 0/4 5/12 7

Visible against silent

Count
Calls made 360
Calls that failed structural validation 0
Cells that diverged 12
of which visible to a schema check 0
of which silent 12

What a single trial is worth

Computed on the 24 cells whose ground truth is cheap and uncontroversial, closed and binary. The open shape is not scored for correctness here.

Overall closed binary
Cells right on 10 of 10 16 / 24 12 / 12 4 / 12
Cells right on 0 of 10 7 / 24 0 7 / 12
Cells partially right 1 / 24 0 1 / 12
P(cell reliable given one random right trial) 0.947 1.000 0.816

Total measured cost of the 360 calls: 0.247 USD.

What the archive contains

Six files, 697994 bytes uncompressed. The inventory is here so you know what you are downloading before you download it.

File What it holds Size
entrees.py builds the 12 inputs from the three source corpora, with their instructions 5.4 KiB
conduire-test.py the 360 calls, interpreting nothing 5.0 KiB
resultats-bruts.json the 360 responses, raw, with the 12 instructions used 624.4 KiB
analyser.py the whole classification, recomputable without calling a model 10.5 KiB
chiffres.json the computed figures, plus one line per cell 24.0 KiB
PROTOCOLE.md the protocol, the reading of the figures, and the limits 12.3 KiB

The scripts and their comments are in French, which is the working language behind this site. The API key is read from an environment variable and appears in none of these files.

Rerun it

The analysis calls no model at all, so it replays at zero cost on the raw responses shipped in the archive. Only conduire-test.py spends anything.

bash
unzip 2026-08-temperature-zero-repeatability.zip
python3 conduire-test.py --sortie resultats-bruts.json --repetitions 10 \
    --modeles openai/gpt-4o-mini google/gemini-2.5-flash-lite anthropic/claude-haiku-4.5
python3 analyser.py --brut resultats-bruts.json --sortie chiffres.json

--rejouer resultats-bruts.json only recalls the lines marked lu: false, so an interrupted run resumes without paying twice.

Each raw response is one entry of resultats-bruts.json, exactly as the model returned it, before any classification. This one is rank 9 of the cell that diverged on rank 10:

json
{"modele": "google/gemini-2.5-flash-lite", "entree": "binaire-carnet-01",
 "forme": "binaire", "rang": 9, "lu": true, "json_valide": true,
 "reponse": {"groups": [["1","10"],["3","8"],["6","9"],["7","11"]],
   "notes": "Row 5 and 4 have the same first name and last name, but different companies and emails, suggesting they are different people."}}

Verify what you downloaded before you trust it:

bash
shasum -a 256 2026-08-temperature-zero-repeatability.zip
# deda8d0f9829ed98807a39b19619cd42f1fe1111b9e25a477551b41deef493fd

How to cite this dataset

Copy one of the two forms below. The reference points at this page, which is the stable address of this dataset; the archive is what that address serves.

citation
Think AI First (2026). Repeatability of an LLM step at temperature 0, 360 raw responses.
Think AI First. Test conducted 2026-08-25, published 2026-09-01.
https://think-ai-first.com/datasets/temperature-zero-repeatability-2026-08. Licensed CC BY 4.0.
bibtex
@dataset{thinkaifirst2026repeatability,
  author    = {{Think AI First}},
  title     = {Repeatability of an LLM step at temperature 0, 360 raw responses},
  year      = {2026},
  publisher = {Think AI First},
  url       = {https://think-ai-first.com/datasets/temperature-zero-repeatability-2026-08},
  note      = {Test conducted 2026-08-25. Licensed CC BY 4.0. Archive SHA-256 deda8d0f9829ed98807a39b19619cd42f1fe1111b9e25a477551b41deef493fd}
}

Licence and reuse

Published under CC BY 4.0. Use it, quote it, contradict it. If you rerun this and get something different, tell us through the contact form: corrections are published on the article itself, dated.

Download the dataset (zip, 37.5 KiB, 6 files)

Other datasets published here

Each one is a separate test, with its own corpus, its own raw responses and its own archive. This one reuses inputs from all three.