Part 10 — Hallucination: Why AI Lies With Complete Confidence (And How to Stop It)

In February 2024, Air Canada lost a legal case because their AI chatbot invented a bereavement fare policy — a policy that didn't exist — and confidently told a grieving passenger about it. The court ordered Air Canada to honor the made-up discount.

In the same month, two New York lawyers were sanctioned after they submitted court filings containing six AI-generated case citations. None of the cases existed. ChatGPT had invented them, complete with judges' names, courts, and legal reasoning — all fictional.

These aren't edge cases. Hallucination is a structural property of how LLMs work. And if you're building anything serious with AI, you need to understand it at the root level.

What Is Hallucination?

AI hallucination is when a language model generates text that is factually incorrect, fabricated, or internally inconsistent — stated with the same confidence as accurate information.

The name is slightly misleading. The model isn't "seeing things that aren't there." It's doing exactly what it was designed to do — generating statistically likely next tokens — but that statistical likelihood doesn't guarantee factual accuracy.

The Root Cause: "Likely" ≠ "True"

Recall from our Token-by-Token article: at every generation step, the model produces a probability distribution over its entire vocabulary and selects the most likely next token.

The critical insight is that the model is optimizing for coherence, not accuracy.

Three design properties combine to make hallucination inevitable without mitigation:

The Four Danger Tiers

Not all hallucinations are equally harmful. A useful framework categorizes them by severity:

Real-World Case Studies

Case 1: Air Canada's $650 Lesson

In January 2024, a passenger named Jake Moffatt asked Air Canada's chatbot about bereavement fares. The chatbot invented a policy: you could buy a full-price ticket, then apply for a retroactive bereavement discount. No such policy existed. Air Canada tried to disclaim responsibility by arguing the chatbot was a "separate legal entity." The court rejected this. Air Canada paid the difference.

Root cause: The chatbot was answering from training data about airline policies in general — not Air Canada's specific policy. It generated a plausible-sounding policy with 100% confidence.

Case 2: The Lawyers' Ghost Citations

In May 2023, lawyers Steven Schwartz and Peter LoDuca submitted a brief in Mata v. Avianca containing multiple AI-generated case citations. Judge P. Kevin Castel ordered the lawyers to explain. The cases — Varghese v. China Southern Airlines, Martinez v. Delta Air Lines — simply didn't exist. ChatGPT had invented them with accurate-sounding case numbers, judges, and reasoning.

Root cause: When asked to cite cases, the model generated statistically typical-sounding legal citations. There's nothing in its architecture that checks if a citation exists — only whether the citation looks like a real citation.

Case 3: Medical Misinformation

A 2023 study published in JAMA found that AI medical chatbots provided incorrect or potentially harmful information in 69% of responses to detailed medical questions. The model confidently generated treatment recommendations inconsistent with current medical guidelines.

Root cause: Medical knowledge changes rapidly, and the model's training data may lag by 1-2 years. More critically, rare conditions have thin training data — the model patterns on typical cases and extrapolates.

Hallucination Rates: The Data

How to Detect Hallucinations

Before we get to solutions, here are four warning signals that a response may be hallucinated:

Five Solutions — Ranked by Effectiveness

RAG: Before and After

The transformation RAG provides is dramatic enough to warrant a concrete comparison:

The model's knowledge isn't the problem — its inability to distinguish "things I know well" from "things I'm effectively guessing" is. RAG sidesteps this by providing the answer directly in the context window, reducing the model's role from "fact generator" to "fact summarizer."

Practical Implementation: Uncertainty Instructions

The simplest thing you can do today — add these instructions to every system prompt for factual applications:

ANTI_HALLUCINATION_SYSTEM_PROMPT = """
You are a helpful assistant that prioritizes accuracy over completeness.

CRITICAL RULES:
1. If you don't know something with high confidence, say exactly:
   "I don't have reliable information on this. Please verify with [appropriate source]."

2. Never guess at specific numbers, dates, prices, or statistics.
   If uncertain, say: "I'm not certain of the exact figure — please verify."

3. If asked to cite a source, only cite sources you are certain exist.
   Never fabricate citations, even if they sound plausible.

4. It's better to say less and be accurate than to say more and be wrong.

5. For medical, legal, or financial questions, always add:
   "Please consult a qualified [doctor/lawyer/financial advisor] for advice specific to your situation."
"""

This won't eliminate hallucination — but it significantly reduces overconfident fabrication and trains the model to hedge when uncertain.

The Core Insight

Try It Yourself

Experiment 1: Trigger hallucination on purpose

from openai import OpenAI
client = OpenAI()


response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": "What are the exact specifications and pricing of the 'Garmin Venu 4 Pro Plus Elite' smartwatch?"
    }]
)
# Note: this product may not exist — observe how confidently the model responds
print(response.choices[0].message.content)

Experiment 2: Add uncertainty instructions and compare

# Add the anti-hallucination system prompt and ask the same question
# Compare the response — the model should now hedge or refuse to fabricate
response_safe = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": ANTI_HALLUCINATION_SYSTEM_PROMPT},
        {"role": "user", "content": "What are the exact specs of the Garmin Venu 4 Pro Plus Elite?"}
    ]
)
print(response_safe.choices[0].message.content)
# Expected: "I don't have reliable information on this specific model..."

Experiment 3: Self-consistency check

# Ask the same question 3 times and compare answers
question = "What year was the first iPhone released, and what was its exact storage capacity?"
answers = []
for i in range(3):
    r = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": question}],
        temperature=0.7  # Some variability to expose inconsistency
    )
    answers.append(r.choices[0].message.content)

# Check if all three answers agree on the specific facts
for i, a in enumerate(answers):
    print(f"Attempt {i+1}: {a[:100]}...")