Skip to main content
TekStudio.ai

AI Foundations Lesson 3 of 5

Why AI Gets Things Wrong

Hallucination is not a glitch to be patched. It is the same process that makes AI useful, running without a fact-check.

Foundation Evergreen 8 min

Worth reading first: What AI Actually Is

By the end of this lesson you will be able to

  • Explain why AI invents plausible-sounding facts
  • Rank tasks by how likely they are to produce a fabrication
  • Apply a verification routine proportional to the stakes

A lawyer once filed a brief citing six cases that did not exist. The AI had produced correct-looking citations — plausible names, plausible years, plausible court. None of them were real. That story gets told as a warning about carelessness. It is more useful as a lesson about mechanism.

Where made-up answers come from

The model generates text that fits the pattern. A legal citation has a very strong pattern: a case name, a versus, a reporter volume, a year. Asked for citations supporting an argument, the model produces things shaped exactly like citations. Whether those particular cases exist is a different question, and nothing in the process asks it.

DefinitionHallucination
Confident output that is not grounded in any real source — invented facts, citations, quotes, or figures produced because they fit the pattern of a correct answer. Full definition
The counter-intuitive partHallucination is not the model malfunctioning. It is the model doing exactly what it does — producing plausible text — applied to a question where plausibility and truth come apart. You cannot remove it without removing the generation itself.

Which tasks are actually risky

Risk is not uniform. It scales with how specific and how checkable a claim is, and with how much of the answer the model had to supply from memory rather than from something you gave it.

TaskRiskWhy
Rewrite this paragraph more clearlyVery lowThe source material is in front of it
Summarise this document I pastedLowGrounded, though it can over-compress
Draft an email about a meeting I describeLowYou supplied the facts
Explain a general conceptMediumUsually right, occasionally confidently dated
What are the current rules for X?HighSpecific, changeable, recalled from training
Give me sources for this claimVery highCitations have a strong pattern and are easy to fake
What is this company's revenue?Very highSpecific number, no grounding
Fabrication risk by task type.

Read down that table and a rule emerges: the further the task is from "here is the material, work with it", the more verification it needs.

The fix that actually works

The single most effective intervention is grounding — making sure the answer is built from real material you supplied rather than from recall. In practice that means pasting the document. At scale, it means a retrieval system that finds the relevant passages and inserts them automatically.

Grounding, step by step
Ask something:

“How much do you charge if I live outside town?”

  1. 1

    Retrieve

    Search the knowledge base for passages that look related to the question.

  2. 2

    Augment

    Paste those passages into the prompt, above the original question.

  3. 3

    Generate

    The model answers using the passages, not its training data.

Step through retrieval. Note the last case — when nothing relevant is found, a well-built system says so rather than filling the gap.
How retrieval finds the right passage
Search for:
  • Billing
  • Scheduling
  • Complaints
  • Hiring

Nearest matches to “money we are owed”

  1. 1Send the invoice 86% match
  2. 2Chase the unpaid bill 86% match
  3. 3Payment is overdue 86% match

Simplified Real embeddings use hundreds of dimensions, not two, and positions are learned rather than placed by hand. The idea is identical: text becomes coordinates, and closeness means similar meaning — which is why search works even when you use none of the same words.

Similar meaning ends up close together, which is why search works without matching words. Try the unrelated query last — retrieval still returns three results.

The same question, asked two ways.

Ungrounded

  • "What is our refund policy?"
  • Model answers from general patterns
  • Produces a plausible, generic policy
  • Sounds right, matches nothing you wrote

Grounded

  • "Using the policy below, what is our refund window?"
  • Model answers from the text you supplied
  • Quotes or paraphrases the real clause
  • Wrong answers become obvious against the source

A verification routine you will actually follow

Advice to "always check everything" gets ignored because it is impractical. Scale the check to the stakes instead.

Before you use AI output for anything that matters:

  1. Every proper noun, number, date, and citation gets checked against a real source. No exceptions in client-facing or legal work.
  2. Anything the model asserted that you did not supply is treated as a claim to verify, not a fact.
  3. If it cited a source, open the source. A correct-looking reference to a document that does not exist is the classic failure.
  4. For anything recurring, spot-check the first ten outputs properly, then sample. You are testing the process, not each result.
  5. If you cannot verify it and it matters, do not ship it.
The reframe worth keepingTreat AI output as a confident draft from a fast, well-read colleague who has never seen your files and will not tell you when they are guessing. You would check that person's numbers too.
Can I just ask it whether it is sure?

No. Asking "are you certain?" produces text about certainty, generated by the same process. Models will often cave and revise a correct answer when pushed, which makes the question worse than useless.

Do newer models hallucinate less?

Generally yes, and that cuts both ways. Fewer, subtler errors are harder to catch precisely because the surrounding output is more reliable. Verification habits matter more as models improve, not less.

What about tools that cite sources as they answer?

A real improvement — a cited claim is checkable. But check that the source says what the summary claims. Citing a real document that does not support the point is a common and easily missed failure.

Key takeaways

  • Hallucination is the normal mechanism applied where plausible and true diverge.
  • Risk rises with specificity and falls with grounding.
  • Supplying the source material is the single most effective fix.
  • Verify proper nouns, numbers, dates, and citations — always, in work that matters.
  • Asking the model whether it is sure tells you nothing.