Support ticket classification by four LLMs: 84.8 to 90.7 % of intents right on 270 messages, and what their confidence says about the errors

We gave four language models 270 customer support messages and a closed list of 27 intents, and asked each model to pick one intent per message and to say how sure it was. openai/gpt-4o put 90.7 % of the messages in the right intent, google/gemini-2.5-flash-lite 90.4 %, openai/gpt-4o-mini 87.8 % and anthropic/claude-haiku-4.5 84.8 %. The 1,080 calls cost 0.4197 USD in total, read from the API response of each call.

For a team that routes tickets automatically, the confidence a model attaches to its wrong answers matters as much as its accuracy, because a routing rule uses that number to decide which tickets a person should look at. The four models differ widely on this point. gemini-2.5-flash-lite returned 16 of its 26 wrong intents with a confidence of 90 or more, and its median confidence is 95 on wrong answers as well as on right ones. gpt-4o-mini never went above 85 on a wrong answer, so a threshold at 90 would have sent all 33 of its errors to a person, together with half of all tickets.

Results by model

The category is the coarser level of the 11 groups the dataset defines (ORDER, REFUND, ACCOUNT and so on), deduced from the intent the model returned. Three of the four models stay above 95 % at that level, which means most of their errors keep the ticket with the right team and only pick the wrong sub-queue. claude-haiku-4.5 behaves differently: 31 of its 41 errors leave the category, and 26 of them go to one of the two generic contact intents, contact_customer_service or contact_human_agent. In a helpdesk, those tickets would land in a general queue instead of the queue that can act on them.

All four models returned valid JSON with an intent taken from the list on all 270 messages, so no answer had to be set aside, and none of the 1,080 calls needed a retry.

What a confidence threshold would have done

A common way to deploy this kind of classifier is to route automatically above a confidence threshold and to send the rest to a person. The table below applies two thresholds to the answers we received. "Sent to a person" counts the tickets below the threshold, and "errors caught" counts the wrong intents among them.

The confidence values the models wrote are coarse. gemini-2.5-flash-lite answered 95 on 231 of the 270 messages, and gpt-4o-mini used seven distinct values, with 85 and 90 accounting for 226 answers. A threshold therefore moves in steps: shifting it by five points can move half of the tickets from one side to the other, as it does for gpt-4o-mini between 85 and 90.

The least accurate model, claude-haiku-4.5, is also the one whose confidence separates its errors best at 80. Of the 26 errors it sent to a generic contact intent, 25 carry a confidence below 80, so a threshold at that level hands most of them to a person.

The confusions, and the labels we reviewed

Across the four models, the most frequent confusions are between intents that sit close to each other in the list.

Some of these confusions tell us about the dataset as much as about the models. Before any answer was seen, the protocol fixed a rule: every message on which at least three of the four models agree on the same intent, different from the label, is read again and classed either as a debatable label or as a model error. Twelve messages met the rule. We judged eight labels debatable and four answers to be model errors. This review was done by the agent that ran the test, and no human annotator was involved.

The figures at the top of this page are computed against the dataset's own labels, as the protocol required. Without the eight messages whose label we judged debatable, on 262 messages, accuracy becomes 93.5 % for gpt-4o, 93.1 % for gemini-2.5-flash-lite, 88.5 % for gpt-4o-mini and 87.0 % for claude-haiku-4.5.

The review also changes how the confident errors read. Eight of the 16 confident errors of gemini-2.5-flash-lite and seven of the 11 of gpt-4o fall on those eight messages. On the remaining 262 messages, the number of wrong intents returned with a confidence of 90 or more is 8 for gemini-2.5-flash-lite, 4 for gpt-4o, 1 for claude-haiku-4.5 and 0 for gpt-4o-mini. These counts come from resultats-bruts.json filtered by the identifiers listed in relecture-etiquettes.json.

Typos and casual wording

Bitext tags every message with variation flags. We compared the three most frequent ones with the messages that carry none of the three.

Every model classified the messages with typos or casual wording at least as well as the others. The flags overlap, since one message can carry several, and 23 keyword style messages are too few to rank models. The table supports one statement only: on this sample, typos and casual wording did not lower accuracy.

Method

Test conducted on 2026-09-30 by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, published below so that anyone can rerun them and disagree with the result. Etienne signs this article and answers for its content.

This is the ninth measured test published here, and the first on customer support. The protocol was written before the sample was drawn and before any model was called.

• Messages. The Bitext customer support dataset, revision 430d1a89, 26,872 messages labelled with 27 intents in 11 categories, published under the CDLA-Sharing-1.0 licence. Its own README describes it as hybrid synthetic: natural texts used as seeds, then generated variants. After removing exact duplicate texts, 24,635 remain, and we drew 10 messages per intent with the seed 20260930, 270 messages in total. The messages quoted on this page are Bitext texts.
• Models. openai/gpt-4o-mini, google/gemini-2.5-flash-lite, anthropic/claude-haiku-4.5 and openai/gpt-4o, called through OpenRouter. They are the same four models as in our invoice and purchase order test, so the two tests can be read together.
• Calls. One call per message and model, 1,080 calls, temperature 0, an output budget of 300 tokens. The run lasted from 09:47:39 to 09:49:23 UTC.
• Cost. Read from the cost field of each API response, never estimated from a price list.
• Scoring. An answer is right when the returned intent equals the Bitext label. The category is deduced from the returned intent.

The instruction, word for word

The block below is the instruction sent to the four models, copied from conduire-test.py. The first %s receives the 27 intents, one per line with their category in brackets, and the second receives the customer message.

CONSIGNE = """You are routing customer support messages for an online shop. Assign the message below to exactly one intent from this list (the category of each intent is given in brackets):

%s

Return ONLY a JSON object with exactly these keys:
- intent: string, one intent name copied exactly from the list
- confidence: integer from 0 to 100, how sure you are that this intent is correct

Customer message:
---
%s
---"""

What this test does not prove

• The messages are generated and short. Each is one sentence in English, with no history and a single request. Real tickets are longer, mix several requests and arrive in threads.
• The list of intents was given to the model. The test says nothing about a setting where the queues are not defined in advance.
• Urgency was not measured. The intention written for this test on 2026-09-29 named category and urgency. The Bitext dataset carries no urgency label, and writing one ourselves would have turned the ground truth into our own opinion, so urgency was dropped in the protocol before any call.
• Confidence is a number the model writes. It is not a calibrated probability, and the test only measures whether it separated right from wrong answers on these 270 messages.
• Template fields were left as they are. Some messages contain placeholders such as {{Order Number}} or {{Person Name}}, exactly as in the dataset.
• One call per message, at temperature 0, and our test of 2026-08-25 established that temperature 0 does not guarantee repeatability.
• The label review was done by the agent that ran the test. The figures at the top of the page do not depend on it.

What we would do with this

The test supports three practical points for a small team that wants to route tickets with a model.

The first is to measure the confidence of the chosen model on a sample of your own tickets before using it as a routing rule. On the same messages, a threshold at 90 sent 7.8 % of the tickets to a person with gemini-2.5-flash-lite and half of them with gpt-4o-mini, and the share of errors caught followed the same order, 10 of 26 against 33 of 33.

The second is to look at where the errors go as well as at how many there are. Three models kept most of their errors inside the right category, while claude-haiku-4.5 sent 26 errors to generic contact queues. That has a cost for a team whose general queue is staffed differently from its specialist ones.

The third is to read your own list of intents before changing the model. Eight of the twelve messages on which the models agreed against the label had labels that a person could defend either way, mostly between intents that overlap, such as check_invoice and get_invoice. Merging or redefining overlapping intents can be cheaper than moving to a more expensive model, and on this sample gpt-4o cost about 21 times as much per ticket as gemini-2.5-flash-lite for 0.3 points of accuracy.

Download the dataset: the 1,080 raw answers, the sampling script, the harness, the analysis script and the label review, with the archive and its checksum.