Part 9 — Context Window: The Invisible Limit That Shapes Every AI Conversation

Last article we saw how the model generates one token at a time inside the Transformer. Today we explore the exact box that limits how much it can see while doing so.

Have you ever noticed that AI seems to handle the beginning and end of a long conversation well, but occasionally "forgets" critical information you mentioned in the middle?

That's not random. It's physics — specifically, the physics of how Transformer attention distributions work. And once you understand it, you'll never structure a long AI prompt the same way again.

What Is a Context Window?

The context window is the total amount of text an LLM can "see" at any one moment. It's not a memory — it's more like a field of vision.

Critical distinction: The LLM has no persistent memory between API calls. Every time you send a message, the API receives:

Think of it like a doctor who reads your entire medical file from scratch at every appointment. They're not remembering — they're rereading. And if the file is too thick for the file box, some pages get left out.

Context Window Sizes: How Models Compare in 2026

An important nuance: Larger context windows don't automatically mean better quality. The "Lost in the Middle" problem means that even models with 1M token windows may miss information buried in the middle. Bigger window ≠ perfect recall.

Problem 1: Your Data Is Bigger Than the Window

The most common context window problem in production: your data simply doesn't fit.

The fundamental insight: you don't need to fit everything in the context window — you need to fit the right things. The vector database handles the "finding the right things" part. The context window receives only the most relevant subset.

Problem 2: Growing Conversation Cost

Every time a user sends a new message, your API call includes the entire conversation history up to that point.

Three strategies to manage conversation growth:

How This Window Relates to Everything We've Learned So Far

The context window isn't an isolated concept — it's the stage where every previous article comes together:

Problem 3: Lost in the Middle — The Most Dangerous Blind Spot

This is the most counterintuitive and impactful context window phenomenon. A 2023 research paper ("Lost in the Middle: How Language Models Use Long Contexts") demonstrated something alarming:

LLMs pay significantly more attention to information at the beginning and end of the context window. Information in the middle gets systematically underweighted. (This is a direct consequence of how self-attention scores are distributed — the model's attention mechanism naturally anchors to the first tokens it sees and the most recent ones, leaving the middle statistically disadvantaged.)

What this means in practice:

The researchers showed that if you have 20 relevant documents in the context window, models answer correctly ~80% of the time when the answer is at the beginning or end — but accuracy drops to ~50% when the answer is in the middle of the document list.

For a 10,000-token RAG response with 10 document chunks, the 5th and 6th chunks may effectively be invisible to the model.

Practical Strategies: Positioning Information to Survive the Dead Zone

Given the Lost in the Middle effect, prompt structure matters as much as prompt content:

Input vs Output: The Pricing Asymmetry

The context window directly determines your API costs, but the split between input and output matters enormously:

Developer tip: Adding "Be concise. Respond in 1-2 sentences unless asked for detail." to your system prompt can cut output costs by 60-80% with minimal quality loss for most use cases.

Context Budget Breakdown: What's Actually Taking Space?

For a typical RAG-powered customer support chatbot with a 10-turn conversation history:

The Core Insight

The numbers are sobering — but the real power comes from treating the window as a stage rather than a dump.

Pro Tips for Builders

Try It Yourself

Experiment 1: Measure your context usage

import tiktoken

def analyze_context(system: str, messages: list, rag_chunks: list, query: str):
    enc = tiktoken.encoding_for_model("gpt-4o")

    system_tokens = len(enc.encode(system))
    history_tokens = sum(len(enc.encode(m["content"])) for m in messages)
    rag_tokens = sum(len(enc.encode(c)) for c in rag_chunks)
    query_tokens = len(enc.encode(query))
    total = system_tokens + history_tokens + rag_tokens + query_tokens

    print(f"System Prompt:      {system_tokens:>7,} tokens ({system_tokens/total*100:.1f}%)")
    print(f"Conversation Hist:  {history_tokens:>7,} tokens ({history_tokens/total*100:.1f}%)")
    print(f"RAG Chunks:         {rag_tokens:>7,} tokens ({rag_tokens/total*100:.1f}%)")
    print(f"User Query:         {query_tokens:>7,} tokens ({query_tokens/total*100:.1f}%)")
    print(f"TOTAL:              {total:>7,} tokens")
    print(f"GPT-4o capacity remaining: {128_000 - total:,} tokens")

analyze_context(
    system="You are a helpful product assistant...",
    messages=[{"role": "user", "content": "Tell me about wearables"},
              {"role": "assistant", "content": "Wearables include..."}],
    rag_chunks=["Ray-Ban Meta specs: ...", "Samsung Ring: ...", "AirPods Pro: ..."],
    query="Which has the best battery life?"
)

Experiment 2: Test Lost in the Middle yourself Create a prompt with 15 facts listed. Put the answer to your question at positions 1, 8, and 15. Ask the model which position it remembers most reliably. You'll see the pattern.

Experiment 3: Measure summarization savings

from openai import OpenAI
import tiktoken

client = OpenAI()
enc = tiktoken.encoding_for_model("gpt-4o")


long_history = [
    {"role": "user", "content": "Tell me about wearables"},
    {"role": "assistant", "content": "Wearables include smartwatches, fitness trackers, AR glasses, and smart rings. Key players: Apple, Samsung, Meta, and Oura."},
    # ... imagine 18 more turns here
] * 10  # rough simulation

history_tokens = sum(len(enc.encode(m["content"])) for m in long_history)
print(f"Full history: {history_tokens:,} tokens")

# Summarize instead
summary_prompt = "Summarize this conversation in 3 bullet points:\n" + \
    "\n".join(f"{m['role']}: {m['content']}" for m in long_history[:6])

summary = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": summary_prompt}]
).choices[0].message.content

summary_tokens = len(enc.encode(summary))
print(f"Summary: {summary_tokens:,} tokens")
print(f"Savings: {history_tokens - summary_tokens:,} tokens ({(1 - summary_tokens/history_tokens)*100:.0f}% reduction)")
print(f"\nSummary:\n{summary}")

Run this and compare the token counts. A 20-turn conversation that costs 8,000 input tokens can often be summarized to under 200 — a 97% reduction with minimal context loss.

Now that you understand the limits of what the model can see, the next article explains why it sometimes confidently makes things up anyway.