Support ticket classification, four models

This page describes one dataset, published in full. It is the raw material behind our measurement of support ticket classification by four models: 270 customer messages, 1,080 answers from four models with the confidence each model declared, and the cost billed for every call.

Everything on this page comes from that one run, conducted on 2026-09-30. Nothing is estimated.

What it measures, and what it does not prove

It measures two things on the same answers: whether a model puts a customer message in the right intent out of a closed list of 27, and whether the confidence it declares separates its right answers from its wrong ones. The category of each answer (11 groups) is deduced from the intent, so the dataset also shows whether an error stays with the right team.

This is the first dataset published here on customer support, and the first whose messages were not generated by our own scripts. They come from a public dataset, which is why their text is handled differently, as explained below.

What it does not prove, stated plainly, because a dataset quoted without its limits does more harm than no dataset at all:

• The messages are hybrid synthetic, by the description of their own publisher, and short: one sentence in English, one request, no history.
• The 27 intents were given to the model. Nothing here applies to open ended triage.
• Urgency is not measured. The source carries no urgency label.
• Confidence is a number the model writes, not a calibrated probability.
• One call per message and model, at temperature 0, and our own test of 2026-08-25 established that temperature 0 does not guarantee repeatability.
• The label review covers twelve messages and was done by the agent that ran the test. The raw figures, against the source labels, are published first and do not depend on it.

Where the messages come from

The 270 messages are drawn from the Bitext customer support dataset, revision 430d1a89bd93bd1fa23c16f29dd53e73f0087443, which is published under the CDLA-Sharing-1.0 licence. Everything else in this archive is ours and published under CC BY 4.0. Because the two licences differ, the message texts are not redistributed: every answer in resultats-bruts.json names its message by a sample identifier, T001 to T270, with the expected intent, the category and the Bitext variation flags.

The sample can be rebuilt exactly. tirer-echantillon.py reads the Bitext file, removes exact duplicate texts, then draws 10 messages per intent with the seed 20260930. We rebuilt it on 2026-10-06 from the source file and obtained the same bytes as the sample the test used, whose sha256 begins with 5f37a8b0. SOURCE-BITEXT.md in the archive gives the full checksum, the exact file and the steps.

The figures, per model

chiffres.json holds these figures with the confusions between intents, the results per variation flag, the effect of confidence thresholds at 80, 90 and 95, and the accuracy computed without the eight messages whose label we judged debatable.

What is in the archive

One raw record, as it is served

Every line of resultats-bruts.json holds one answer. This one is a wrong intent returned with a confidence of 95, shown here without its brut field, which holds the model's text as received.

{
 "modele": "google/gemini-2.5-flash-lite",
 "message": "T026",
 "attendu": "change_shipping_address",
 "categorie_attendue": "SHIPPING",
 "marqueurs": "BL",
 "lu": true,
 "essais": 1,
 "json_valide": true,
 "intention": "set_up_shipping_address",
 "intention_brute": "set_up_shipping_address",
 "categorie": "SHIPPING",
 "confiance": 95,
 "juste": false,
 "categorie_juste": true,
 "raison_d_arret": "stop",
 "jetons_entree": 342,
 "jetons_sortie": 30,
 "cout_usd": 4.62e-05,
 "fournisseur": "Google",
 "modele_servi": "google/gemini-2.5-flash-lite"
}

Verify what you downloaded before you trust it:

shasum -a 256 2026-09-support-ticket-classification.zip
# 0b6cfbee6ddac1be442d07cd1c9e414690df830e431be7dc0117549951c653b7

How to cite this dataset

Copy one of the two forms below. The reference points at this page, which is the stable address of this dataset; the archive is what that address serves.

Think AI First (2026). Support ticket classification by four LLMs, 1,080 raw answers with declared confidence.
Think AI First. Test conducted 2026-09-30, published 2026-10-06.
https://think-ai-first.com/datasets/support-ticket-classification-2026-09. Licensed CC BY 4.0.

@dataset{thinkaifirst2026tickets,
  author    = {{Think AI First}},
  title     = {Support ticket classification by four LLMs, 1,080 raw answers with declared confidence},
  year      = {2026},
  publisher = {Think AI First},
  url       = {https://think-ai-first.com/datasets/support-ticket-classification-2026-09},
  note      = {Test conducted 2026-09-30. Licensed CC BY 4.0. Archive SHA-256 0b6cfbee6ddac1be442d07cd1c9e414690df830e431be7dc0117549951c653b7}
}

Licence and reuse

Published under CC BY 4.0, except the twelve Bitext messages quoted in chiffres.json, which remain under CDLA-Sharing-1.0. Use it, quote it, contradict it. If you rerun this and get something different, tell us through the contact form: corrections are published on the article itself, dated.

Download the dataset (zip, 35.0 KiB, 8 files)

Other datasets published here

Each one is a separate test, with its own corpus, its own raw responses and its own archive. This one is the first on customer support.

• Invoice extraction by three low cost LLMs, 288 raw responses, test conducted 2026-08-10.
• Contact list deduplication by three low cost LLMs, 96 raw responses, test conducted 2026-08-17.
• Meeting action item extraction by three low cost LLMs, 96 raw responses, test conducted 2026-08-18.
• Repeatability of an LLM step at temperature 0, 360 raw responses, test conducted 2026-08-25.
• Four error detection guardrails on 288 invoice extractions, 288 raw self-check responses, test conducted 2026-09-04.
• Price against accuracy on invoice extraction, six models, 864 raw responses with their billed cost, test conducted 2026-09-06.
• Two error detection guardrails on 368 proposed contact merges, 96 raw self-check responses, test conducted 2026-09-12.
• Invoice against purchase order matching by four LLMs, 672 raw decisions, test conducted 2026-09-14.

The same results, by model

The figures of this dataset are also gathered by model, with those of the other tests on this site: Claude Haiku 4.5, Gemini 2.5 Flash Lite, GPT-4o mini. The same inputs are also compared two models at a time: Claude Haiku 4.5 vs Gemini 2.5 Flash Lite, Claude Haiku 4.5 vs GPT-4o mini, Gemini 2.5 Flash Lite vs GPT-4o mini.