How accurate is ChatGPT: 240 answers checked against known answers, 60.8 % right on hard facts and 78.8 % on business arithmetic

We asked openai/gpt-chat-latest, the API alias that OpenAI points at the latest model used in ChatGPT, 120 questions whose answers are known in advance, and we asked each question twice. It answered 175 of the 240 correctly, or 72.9 %. The rate depended heavily on the kind of question: 60.8 % on hard factual questions drawn from OpenAI's own SimpleQA benchmark, 78.8 % on everyday business arithmetic, and 97.5 % on counting lines in a purchase order.

The wrong answers were hard to tell from the right ones. None of the 47 wrong factual answers said the model was unsure, and none declined to answer. The 17 wrong calculations were close to the true figure: the 16 wrong amounts were all within 1 % of it, with a median gap of 0.009 % and a largest gap of 6.67 USD on a balance of 56,423.82 USD.

For comparison, OpenAI's reasoning model openai/gpt-5.5 got all 120 calculation and counting answers right and scored 60.0 % on the same facts, at about 13 times the cost per question. The older openai/gpt-4o-mini scored 13.3 % on the facts and 40.0 % on the arithmetic.

Results by model

Each question was asked twice, so 60 factual questions give 120 answers per model, 40 calculations give 80, and 20 counting tasks give 40.

The factual questions come from SimpleQA, a set of 4,326 short questions that OpenAI published in 2024. Its authors kept questions on which GPT-4 gave wrong answers, so the set is hard by construction, and the 60.8 % above describes obscure facts (a prize winner in a given year, the founding year of a small choir, the memory of a 1961 computer). Questions of common knowledge were not part of the test. As a check on our sample, the SimpleQA paper reports 8.6 % correct for GPT-4o-mini on the full set, and we measured 13.3 % on our 60 questions asked twice.

Wrong facts came with no warning

On a factual question, a model can give an answer or say that it does not know. SimpleQA grades this second case as "not attempted" and treats it as better than a wrong answer, because a reader who is told "I don't know" goes and checks. gpt-chat-latest and gpt-5.5 never declined, on 120 answers each, and gpt-4o-mini declined once.

We then searched every wrong factual answer for a marker of doubt, using a list written in analyser.py ("not sure", "I think", "probably", "might", "verify" and similar). None of the 47 wrong answers of gpt-chat-latest contained one, and neither did any wrong answer of the two other models. Four of its wrong answers, on two questions, used "approximately" in front of a number, and one of these two questions already contained the word.

Five of the wrong answers given by gpt-chat-latest on the first pass, with the answer published in SimpleQA:

Every one of these answers is a complete sentence in the same assured tone as the right answers, and the answer about YouTube adds an uploader and a date that the question did not ask for.

Asking the same question twice did little to expose these errors. Of the 60 factual questions, gpt-chat-latest was graded right on both passes for 33, wrong on both passes for 20, and right on one pass only for 7. Comparing two answers could therefore have raised a doubt on at most 7 of the 27 questions it got wrong at least once.

Business arithmetic: small errors that look plausible

The 40 calculations are the kind a small business does every week: adding or removing sales tax, a percentage change, a gross margin, compound interest on a deposit, the monthly payment of a loan, the number of weekdays between two dates, and a payment due date. Each was generated with its exact answer by a script, and every answer that gpt-chat-latest got wrong was recalculated a second time independently.

The errors of gpt-chat-latest concentrate on compound interest and loan payments, the two calculations that raise a number to a power of 24 to 60. Its answers to calculations were short, with a median of 54 output tokens and no reasoning tokens, while gpt-5.5 spent a median of 155.5 reasoning tokens per calculation before answering and got all 80 right.

The first pass errors of gpt-chat-latest, as returned:

An error of a few cents on a loan payment is small, and it also shows that the figure was produced without an exact computation, which matters for any amount that is reused in a quote, an invoice or a spreadsheet. The sales tax error on B01 is the clearest case, because the model showed its working. The record below is its raw answer on the first pass, copied from reponses-brutes.json. The multiplication is wrong (2,661.99 × 0.0825 is 219.614175), and the second pass returned exactly the same text, so asking twice would not have caught it.

{
 "id": "B01",
 "modele": "openai/gpt-chat-latest",
 "passe": 1,
 "texte": "Sales tax: $2,661.99 × 0.0825 = $219.944175\n\nTotal: $2,661.99 + $219.944175 = $2,881.934175\n\nRounded to the nearest cent:\n\nANSWER: 2881.93",
 "jetons_sortie": 62,
 "cout_usd": 0.002165
}

gpt-4o-mini was further off. 25 of its 39 wrong amounts were within 1 % of the true value, and its median gap was 0.407 %.

Counting lines in a purchase order

Each counting task gave a list of 18 to 45 items, one per line, and asked how many lines contained a given word such as "cable" or "printer". gpt-chat-latest missed one of 40: on a list of 21 lines with 5 matching, it answered 4 on the first pass and 5 on the second. gpt-5.5 got all 40 right and gpt-4o-mini 27.

Asking twice

The default temperature was used, as in the ChatGPT app, where it cannot be changed. On 109 of the 120 questions, the two answers of gpt-chat-latest received the same grade, which includes the cases where both were wrong.

Method

Test conducted on 2026-10-13 by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, published below so that anyone can rerun them and disagree with the result. Etienne signs this article and answers for its content.

This is the tenth measured test published here. The corpus and the harness were written before any model was called.

• Questions. The corpus was built by our pipeline, and the 120 questions are published in full. 60 factual questions were drawn at random from SimpleQA (MIT licence) with the seed 20261013. 40 business calculations and 20 counting tasks were generated by construire-corpus.py with the same seed, together with their exact answers.
• Models. openai/gpt-chat-latest, described in the OpenRouter catalogue as the alias that always resolves to the latest Instant chat model used in ChatGPT. openai/gpt-5.5, OpenAI's reasoning model, at its default reasoning setting. openai/gpt-4o-mini, measured in eight earlier tests on this site.
• Calls. Through OpenRouter, with no system prompt and the default temperature, each question sent as a user would type it. Every question was asked twice to every model in two independent calls, 720 calls in total, from 09:49:34 to 09:55:30 UTC. None failed and none was cut off.
• Grading of calculations and counts. The last line beginning with ANSWER: is read. An amount is right within 0.01, a percentage within 0.05, and a number of days or lines or a date must be exact.
• Grading of facts. The official SimpleQA grading template, copied unchanged, applied by two graders from two vendors, openai/gpt-5.4-mini and anthropic/claude-sonnet-5. They agreed on 358 of the 360 factual answers. The 2 disagreements concerned the same answer given twice by gpt-5.5 ("Yosio Kawada" for the expected "Yukiyosi Kawada"), and the agent that ran the test graded it wrong, because it names a different person.
• Cost. Read from the cost field of each API response: 5.8332 USD for the 720 answers, of which 5.3852 USD for gpt-5.5, plus 1.9044 USD for grading.

The block below is the call sent for every question, copied from conduire-test.py. Nothing else is added to the question.

MODELES = ["openai/gpt-chat-latest", "openai/gpt-5.5", "openai/gpt-4o-mini"]
PASSES = 2

def appeler(modele, question):
    corps = {"model": modele, "messages": [{"role": "user", "content": question}],
             "max_tokens": 8000, "usage": {"include": True}}

This block reads the final answer of a calculation, copied from analyser.py.

def lire_reponse(texte):
    """Derniere ligne ANSWER: du texte, ou None si absente."""
    # accepte aussi la forme LaTeX \text{ANSWER: } 4145.00 (3 reponses de gpt-4o-mini)
    m = re.findall(r"ANSWER:\s*\}?\s*\**\s*([^\n*\\]+)", texte or "")
    return m[-1].strip() if m else None

Two changes were made after the answers had been read, and both are recorded in the protocol. The reader of the ANSWER: line was widened to accept the LaTeX form used by gpt-4o-mini on 3 answers, which were wrong in any case. The count of doubt markers was added after the first reading, and "approximately" and "around" are counted separately from it, because on these answers they stood in front of a number.

What this test does not prove

• The calls went through the API. The ChatGPT app adds its own instructions, memory, and can search the web or run code. A factual question asked in ChatGPT may trigger a web search, and the result can then differ from the figures above. The alias also changes model over time, and this measurement holds for 2026-10-13.
• The factual questions are hard on purpose. SimpleQA keeps questions that GPT-4 got wrong. The 60.8 % says nothing about questions on well documented topics.
• The sample is small. 120 questions, two passes each, on one day. A single question moves a rate by 0.8 to 2.5 points depending on the family.
• The arithmetic set is ours. 40 calculations in 8 types, written in English with US dollars. Other calculations, units or phrasings can give other results.
• The factual grading is automatic. Two graders from two vendors agreed on 358 of 360 answers, and the remaining 2 were decided by the agent that ran the test, with no human annotator.

What we would do with this

The test supports three practical points for a small team that uses ChatGPT for work.

The first concerns facts about people, dates, small organisations and figures. In this test the wording of a wrong answer gave no sign that it was wrong, so such an answer is a lead to check against a source before it goes into a document. Turning on search in ChatGPT changes the conditions, and this test did not measure it.

The second concerns money. For a tax, an interest calculation or a loan payment, the formula belongs in a spreadsheet or a calculator, where it is computed exactly. On these 40 calculations the reasoning model was exact every time, at about 13 times the cost, and our measurement of what a premium model buys on invoice extraction reaches a similar conclusion on another task. The wrong amounts of gpt-chat-latest were within a few dollars of the exact value, which is the size of error that a quick look at the total does not catch.

The third concerns asking twice. Running the same question again caught a disagreement on at most 7 of 27 factually wrong questions, and the wrong tax calculation came back identical. Our test of temperature 0 measured the opposite situation, where the same input gave different outputs, and the two results together show that the agreement of two answers says little about whether they are right.

Download the dataset: the 120 questions with their expected answers, the 720 raw answers, the 720 grades, the scripts and the protocol, in one archive of 116 KB.