Part 8 — Token-by-Token: Why AI Generates Text One Word at a Time (And Why It Costs 4x More)

Last article we saw the Transformer architecture. Today we watch it in action during live generation — and discover why the output side is 4× more expensive. Here's something that surprises most developers: ChatGPT doesn't think its answer in advance and then display it. It predicts one token, then another, each prediction using the previous ones as context. It's not writing — it's recursively predicting. The Transformer reads input in parallel, but generation flips that on its head: it's forced to be sequential, because each new token depends on the last.

The One Thing an LLM Does

Strip away the magic, and a Large Language Model does exactly one thing: given all the tokens it has seen so far, predict the single most likely next token.

Think of it like predictive text on your phone — except instead of suggesting 3 words, it chooses from 100,000+ possible tokens, thousands of times, to build a complete response.

Step-by-Step Generation

Let's trace a real example. Prompt: "What's the best smart glasses?"

The final response — "Ray-Ban Meta Ultra — lightweight, 48MP camera, translates 40 languages, full-day battery" — was assembled from sequential predictions. It was never computed all at once.

Autoregressive Generation: The Mathematical Reality

The formal name is autoregressive generation — each output token becomes part of the input for the next prediction. (It's the same next-token prediction the training loop taught the model, now happening live at inference.) This creates a critical asymmetry:

This is why output tokens cost 4× more than input. Reading your 10,000-token prompt can be largely parallelized. But generating each output token requires a sequential forward pass through the full model — there's no way to batch it without changing the output.

The Probability Distribution: Every Token Is a Vote

At each step, the model doesn't just know the one "right" answer. It produces a probability distribution over its entire vocabulary — every possible next token with a likelihood:

This is the same softmax activation we saw inside the neuron and the Transformer block — here it converts raw logits over the full vocabulary into a distribution over what to say next. The model selects one token, appends it, and runs the whole process again until it emits an [END] token or hits the max length.

KV Cache: The Hidden Optimization That Makes This Workable

Here's the obvious problem: if each new token required the model to re-read the entire context, compute time would grow quadratically. A 1,000-token response from a 10,000-token prompt would be impossibly slow. The solution is the KV Cache (Key-Value Cache):

During the Transformer's attention computation, every token produces a Key and Value vector that don't change once computed. The cache stores them, so each generation step only computes K/V for the new token. This reuse is only possible because of the self-attention mechanism from the last article — without Q/K/V there'd be nothing to cache. It's also another reason reading input is cheaper than generating output: input is one parallel forward pass, output is one-at-a-time even with the cache.

TTFT and Throughput: The Two Metrics That Matter

When building AI apps, "speed" is two different numbers:

The Real Cost Impact: A Developer's Calculator

Since output is 4× more expensive, how you instruct the model affects your bill more than how much data you send. Scenario: 1,000 API calls/day on GPT-4o.

Same task. Same quality. 5× cost difference — just by controlling output length in your prompt.

Streaming: The UX Secret

Since the model generates token-by-token anyway, streaming is free — show each token as it's produced instead of waiting for the whole response.

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What are the best smart glasses in 2026?"}],
    stream=True,      # ← all you need; free, transforms UX
    max_tokens=150,   # ← control output length = control cost
)
for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

This is literally how ChatGPT's web interface works — the streaming appearance is the model's natural behavior surfaced directly.

The One-Way Street: Generation Is Irreversible

The most important practical implication: the model cannot go back and correct a previous token. Once "Ray" is generated, every subsequent token is conditioned on it. The model can't reconsider.

Three takeaways: prompt quality matters more than you think (an ambiguous prompt can commit the model to the wrong interpretation early); this is why hallucination is hard to fix (the model doesn't "realize" a mistake — it just stays consistent with it); and output-format instructions help because they shape the first-token probability distribution.

Developer Quick Reference

The Core Insight

Pro Tips for Builders

Try It Yourself

Estimate cost before you build:

import tiktoken

def estimate_cost(prompt: str, expected_output_words: int) -> dict:
    enc = tiktoken.encoding_for_model("gpt-4o")
    input_tokens = len(enc.encode(prompt))
    output_tokens = int(expected_output_words * 1.33)   # ~0.75 words/token
    input_cost = (input_tokens / 1_000_000) * 2.50
    output_cost = (output_tokens / 1_000_000) * 10.00
    return {
        "total_per_call": f"${input_cost + output_cost:.6f}",
        "daily_1000_calls": f"${(input_cost + output_cost) * 1000:.2f}",
    }

print(estimate_cost("You are a helpful assistant. What are the best smart glasses?", 200))
# → {'daily_1000_calls': '$2.67', ...}

Then measure TTFT live by timestamping the first streamed chunk, and compare stream=True vs stream=False to feel the UX difference — same total time, very different perception.

Key Takeaways

Up Next in the Series