Ordinary software breaks loudly. A language model does something stranger: it produces a confident, well formed answer whether or not it has one. Here is where that bites, and what catches it.
When ordinary software cannot do something, it says so. A spreadsheet formula pointed at text returns an error. A script that cannot open a file stops with a message. The failure is loud, immediate, and impossible to mistake for success.
A language model does not behave this way, and understanding why is most of what you need to use one safely at work.
A generative model produces the continuation that best fits the pattern of what you gave it. That is the whole operation. It is not looking anything up, and it has no separate step where it checks whether an answer exists before producing one.
So when you ask something it cannot answer, it does not stop. It produces the thing that has the shape of an answer, because producing well shaped output is exactly what it was trained to do. Fluency and correctness are different properties, and only the first one is guaranteed.
This is why the word hallucination misleads. It suggests a glitch in an otherwise reliable memory, an occasional fault. There is no memory to fault. The system is working normally in both cases, and it cannot tell you which case you are in, because from the inside they are the same operation.
The plausible reference. A citation, a standard, a document name, a section number. These have a very regular shape, which makes them easy to produce and hard to spot as invented. They pass the eye test precisely because they look correct.
The reasoning that holds together and lands wrong. Each step follows from the last, the argument is coherent, and the conclusion is false because an early premise was invented. Coherence is not evidence here, and it is very persuasive to a reader who is skimming.
The near miss on your own material. You paste a document and ask for a summary. The summary is accurate on the parts you remember, so you trust it, and it quietly misses the one clause you had not read carefully. Trust built on the checkable parts transfers to the unchecked parts, which is exactly backwards.
The unexpected input shape. A step that works on well formed inputs receives a malformed one. Instead of failing, it produces something plausible from what it can make of it, and passes it on. Nothing in the chain signals a problem.
The phrasing sensitivity. Asking the same question two different ways can produce two substantively different answers. Neither is flagged as less certain. If you asked once, you have no idea which one you got.
The reflex is to write better prompts. Prompting helps with output shape, and it does not create knowledge that was never there. These specific failures need checks, not phrasing.
Check against something outside the model. For anything factual, the reference, the figure, the clause, the check has to come from a source that is not the thing being checked. Asking the model whether it is sure is not a check: it will produce a confident answer about its confidence in exactly the same way it produced the original.
Verify the parts you cannot remember, not the parts you can. This inverts the natural instinct. You skim the familiar sections and trust the rest. Do the opposite: the unfamiliar material is where an error can survive.
Ask twice, differently. For anything consequential, put the question in two different framings. Two answers that disagree tell you the ground is soft. It costs a minute and it is the cheapest reliability check available.
Constrain the shape and validate it. Where output feeds another step, require a fixed structure and check the structure before continuing. Free prose arriving in a field meant for a value is a failure you can detect automatically, which is better than one you detect in a month.
Give the step permission to say no. A task that must always produce an answer will produce one. Where you can, make no answer an acceptable output, and route it to a person. This single change converts a class of silent errors into a small visible queue.
After using one of these tools on real work for a while, most people develop a feel for where it drifts on their particular kind of material. That instinct is genuinely valuable and it is worth deliberately building: keep the outputs you rejected and look at them as a group. The pattern in your rejects is a map of where this tool fails on your work.
But the instinct has a limit that is worth naming. It is built from the errors you caught. The errors you did not catch are, by definition, absent from it, and those are the ones that matter most. That is why the checks stay in place even after you have got good at spotting problems, and why the tasks worth automating remain the ones where being wrong is cheap and visible.
Most advice starts with a tool and looks for a use. Start with the task instead, and put it through three questions. Two of them are about checking, and that is the point.
Read the articleOne email when a new article is published. No tool of the week, no affiliate list, no forwarding of your address to anyone.
