Hallucination
Why it matters
For a business, the risk is not embarrassment in a chat window. It is an agent that quotes the wrong price, cites a policy that does not exist or invents a number in a report, and costs trust or money before anyone notices. Hallucination can be reduced but not removed, so the aim is to manage it. Better prompting helps a little. Grounding and checking help far more.
How to apply it
- Ground answers in real documents where stakes are high, using retrieval-augmented generation instead of trusting the model's memory.
- Ask for a source you can verify, then verify it. A confident citation is not proof that the source exists.
- Put a person on anything that touches a price, policy or customer detail before it goes out.
- Treat a suspiciously precise number or quote as a reason to check, not as extra credibility.
- Never let an unchecked agent take an irreversible action based on a fact it was not grounded in.
What it is
A language model does not look facts up. It predicts the most plausible next words given what it has read. Usually the most plausible continuation is also true. Sometimes it is not, and the model has no internal alarm to tell the difference. The result reads exactly like a correct answer: same tone, same confidence, same polish.
Typical examples are a quoted price that was never charged, a court case or study that does not exist, a function in a software library that has never been part of it, and a customer detail blended from two different customers.
Common mistakes
- Assuming a better model has solved it. Newer models hallucinate less, but none are immune. Design the process as if some wrong answers will get through.
- Treating a citation as proof. A confident reference, with an author and a year, can be invented. Open the source and check that it says what the model claims.
- Checking everything equally. Reviewing every sentence wears people out. Focus checks on prices, policies, names, numbers and quotes.
- Asking the model whether it is sure. It will often confirm its own answer with the same confidence. Check against a source outside the model.
- Letting the agent act on ungrounded facts. The cost rises sharply when a made-up fact triggers an action, such as an email sent or a record changed.
