We asked the model behind ChatGPT 120 questions with known answers, twice. It got 60.8 % of hard facts and 78.8 % of business calculations right, and none of its wrong answers said it was unsure.
Read the articleFour models sorted 270 customer support messages into 27 intents. gpt-4o got 90.7 % right and claude-haiku-4.5 84.8 %. gemini-2.5-flash-lite returned 16 of its 26 errors with a confidence of 90 or more, and gpt-4o-mini returned none.
Read the articleWrong price, line never ordered, wrong reference, wrong currency: caught 383 times out of 383. All seventeen wrong approvals of the test sit on one condition, over billing, and one model wrote the discrepancy down before approving it.
Read the articleSelf-checking catches 97.6 % of bad invoice extractions. On contact merges, where nothing can be recomputed, it catches 76.0 %. Cross-model agreement drops from 92.9 % to 62.9 %, and it works worst on the model that makes the fewest mistakes.
Read the articleOn clean invoices the upgrade buys nothing: all six models are at 100 %. Where it does buy something, all three vendors buy the same 10 or 11 correct invoices out of 96, and the price of that identical gain varies by a factor of 6.5. Costs read from the API, never from a price list.
Read the articleThe arithmetic check we published in August catches 59 % of the errors and fires on 22 % of the correct extractions. We put four different guardrails on the same 288 extractions. The one that catches the most is the worst one to deploy, and cross-model agreement never fires on a clean invoice.
Read the articleThe same input, the same model, the same instruction, ten times at temperature 0. A third of the pairs came back with more than one answer, and every single divergence passed a schema check. One model out of three was stable everywhere, which breaks the rule you were about to quote.
Read the articleThree cheap models, 8 meetings, four transcript formats. They find 94.8 percent of the real actions. What they add is the problem: work cancelled during the meeting comes back on one list in five, with somebody's name on it.
Read the articleThree cheap models, 8 address books, four levels of mess. Finding duplicates is the easy part, 94.1 percent. The number that costs you a record is how often two different people get fused with nothing to warn you.
Read the articleThree cheap models, 24 invoices, four levels of messy text. Accuracy is the easy number. The one that costs money is how often a wrong extraction comes back with no warning at all.
Read the articleJobs are bundles of tasks, and bundles rarely disappear at once. Asking about tasks instead produces answers you can actually check, and a clearer view of who decides what happens next.
Read the articleOne sentence explains the technology, and the same sentence explains almost every strange behaviour people run into. The confusion comes from the words we use, not from the machine.
Read the articleBuilding a chain that works is the easy part. Building one whose failures are visible is the actual job, because a model in the middle of a workflow does not stop when it is lost.
Read the articleA small team cannot afford a pilot programme or a tool evaluator. So the decision has to be arithmetic. Here is the calculation, including the cost everyone forgets.
Read the articleOrdinary software breaks loudly. A language model does something stranger: it produces a confident, well formed answer whether or not it has one. Here is where that bites, and what catches it.
Read the articleMost advice starts with a tool and looks for a use. Start with the task instead, and put it through three questions. Two of them are about checking, and that is the point.
Read the articleOne email when a new article is published. No tool of the week, no affiliate list, no forwarding of your address to anyone.
