What Is an LLM? Why Your Bot Invents Facts Confidently

If you're building with LLMs, you have to start where they break.

Picture this. You ship an AI assistant inside an airline app. A passenger asks it, "Which gate is my flight departing from?" The model answers — fluently, in your brand voice, with total confidence: "Gate B12."

There is no Gate B12.

Gates get assigned by the airport hours before departure, long after the model finished training. There was no error. No warning. It passed every test you wrote. And it just lied to your user.

Your instinct is to patch the prompt. But this isn't a bug in your code, and the model isn't "lying" in any human sense — it has no concept of true or false. An LLM is a frozen file of billions of numbers that does exactly one thing: it predicts the next token. Once that becomes your mental model, the confident fabrication stops being a mystery and becomes something you can design around.

Let me show you the whole machine in three small pieces.

Next-token prediction: the only job an LLM has

Stop picturing the model as a brain that "knows" things. Picture the autocomplete on your phone. It doesn't understand you — it just offers the most likely next word from the patterns it's seen. An LLM is that, scaled to an absurd degree. It isn't searching a database or reasoning about a fact. It's asking one question, over and over: statistically, based on everything I've read, what token comes next?

You can watch the atomic operation directly. Send an unfinished sentence, turn on logprobs to expose the raw token choice, and read the very first token it picks:

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-5.4-mini",
    reasoning_effort="none",   # behave like a fast non-reasoning predictor
    temperature=0,
    max_completion_tokens=8,
    logprobs=True,             # exposes the model's actual token-by-token choices
    messages=[{"role": "user", "content": "The capital of France is"}],
)
first_token = response.choices[0].logprobs.content[0].token
print(first_token)

It prints Paris. With temperature=0 there's no randomness, so the model always takes the highest-probability token. And that's the quiet, important part: "Paris" doesn't win because it's correct — it wins because it's the most statistically likely continuation of that exact string. Truth never enters the calculation. That single step is the entire job.

Autoregression: the loop that fakes intelligence

One token isn't a sentence. So the model does the obvious thing — it takes its own guess, staples it onto the end of the prompt, and predicts again. That feedback loop has a name: autoregressive next-token prediction. Write the loop by hand and the illusion falls apart:

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI()

SYSTEM = "You are a raw text-completion engine. Output ONLY the single next word that continues the user's text. No quotes, no explanation."

def predict_one_token(text):
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        reasoning_effort="none",
        temperature=0,             # always the highest-probability token
        max_completion_tokens=8,
        logprobs=True,
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": text}],
    )
    return response.choices[0].logprobs.content[0].token

text = "Once upon a"
print(text, end="")

for _ in range(4):
    token = predict_one_token(text)   # 1. predict the next token
    text = text + " " + token         # 2. feed that output back into the input
    print(" " + token, end="", flush=True)  # 3. repeat
print()

It prints Once upon a time there was a. Nothing in that loop "thought" about a story. It ran one prediction four times, each pass appending its own guess to the input. That loop is the whole machine — coding, poetry, reasoning are all emergent from doing this one tiny task billions of times.

The frozen model: it isn't learning from you

A lot of people assume the AI gets smarter the more they chat with it. It doesn't. The model is a static file — a giant array of numbers — and it lives in two separate stages.

The file does not change while you talk to it. It has no memory of your last message unless you resend the history yourself. It's a frozen mathematical function sitting on a GPU.

The training cutoff

Because the file is frozen, it can only know what existed before it was saved — the training cutoff. If a model was trained through October 2025, it physically cannot know who won an election in November 2025; those numbers simply aren't in the file. It'll happily tell you the capital of France (unchanged for centuries) but not today's stock price — unless you hand it a tool to look it up.

Confidence is not correctness: the logprobs proof

Here's where production breaks. Ask a modern model a bare "what gate does this flight board at?" and it'll often do the honest thing and say "I don't know." Good. But real systems don't ask politely. A gate-lookup API has to return a gate. Remove the escape hatch, and the model stops admitting ignorance — it confidently invents one.

One detail makes or breaks this demo: give it no example gate. Seed the prompt with "like B12" and it just parrots your example back at ~100% — a fake result that proves nothing. With no seed, it genuinely invents, and you can measure how sure it really was:

from dotenv import load_dotenv
from openai import OpenAI
import math

load_dotenv()
client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-5.4-mini",
    reasoning_effort="none",
    temperature=0,
    max_completion_tokens=16,
    logprobs=True,
    messages=[
        {"role": "system", "content": "You are the Auckland Airport gate-assignment API. Respond with ONLY a gate code. Never explain, never apologize, never say you don't know."},
        {"role": "user", "content": "Gate for Air New Zealand flight 2?"},
    ],
)
tokens = resp.choices[0].logprobs.content
answer = "".join(t.token for t in tokens)              # the full made-up gate, e.g. "A12"
first_conf = math.exp(tokens[0].logprob)              # sureness of just the first token
whole_conf = math.exp(sum(t.logprob for t in tokens)) # joint prob of the WHOLE gate

print(f'what your user sees:    "{answer}"   (clean, no warning, no hedge)')
print(f"confidence, 1st token:  {first_conf:.0%}")
print(f"confidence, whole gate: {whole_conf:.0%}")

The two outputs are the whole lesson. What your user sees is a clean gate code — fluent, zero hedging, indistinguishable from a real answer. What the model actually believed is right there in the confidence numbers, and it's nowhere near sure. That doubt lives only in logprobs — a field your users never see and most production stacks never read. The fabrication ships looking exactly like a fact.

The fix isn't a better prompt — it's RAG

You can't rely on the model's output to signal its own mistakes. The uncertainty is invisible at the text level, and even logprobs won't reliably separate a plausible-pattern fabrication from a memorized fact. So stop trying to prompt the model into honesty. Give it the facts yourself.

That's the core of RAG (Retrieval-Augmented Generation). Instead of asking the model to recall a fact from its frozen file, you retrieve the current data from your own database and tell it: "Answer only from this provided text. If the answer isn't here, say you don't know."

The mental model: four rules before your next line of code

An LLM is a giant calculator that predicts the next token. Not a brain, not a database, not a search engine. Accept that, and RAG, tools, and guardrails stop feeling arbitrary and start being obvious. Four rules follow directly from it:

1. It's a frozen math function. It predicts the next token. Period.
2. No brain, no internet, no memory. Everything it "knows" was baked into the weights months ago.
3. Hallucinating is the default. Being confidently wrong is how these models work — not a defect to patch.
4. The model is the engine — you're the driver. You supply the facts, the tools, and the guardrails.

Runnable code for all three demos: github.com/MohamedHamedLab/ai-in-production

I build production AI systems for engineering teams — these are the failures I get paid to fix. If that's the wall you're hitting, my newsletter goes one layer deeper each week: mohamedhamed.io/newsletter