Part 7 — The Transformer: The Architecture That Accidentally Changed the World

Last article we saw how the four learning types and the training loop built ChatGPT. Today we open the box and look at the exact architecture that made all of it possible.

June 2017. Eight researchers at Google Brain sat down and asked a dangerous question: "Why do we even need the RNN?" Then they deleted it. The paper they published — "Attention Is All You Need" — was not patented. It was released freely, and that single decision launched ChatGPT, Claude, Gemini, Llama, and every significant language model that exists today.

Before 2017: The World Ran on RNNs

To understand why the Transformer was revolutionary, you need to understand what it replaced. The dominant architecture for language was the Recurrent Neural Network (RNN). The idea was elegant: read text the way humans do — one word at a time, remembering what came before.

The glasses → remember → are → remember → light → ... → but their battery...

By the time it reaches "battery," the beginning of the sentence has almost completely faded. The RNN had three fatal problems that held AI back for over a decade.

Problem 1: Memory Decay

The RNN maintained a "hidden state" — a compressed memory updated with each new word. The trouble: each update overwrote part of the previous memory. Watch how much of the word "glasses" survives as the sentence goes on:

By the time it reaches "full day," it has forgotten the sentence started with "glasses." Engineers tried to fix this with LSTMs (1997). They helped, but didn't solve it — long documents remained an unsolvable challenge.

Problem 2: Sequential Processing (Speed)

RNNs are inherently sequential. Word 2 can't be processed until Word 1 is done; Word 3 waits for Word 2.

Problem 3: Long-Range Dependencies

In "The glasses are red," the word "red" clearly refers to "glasses." But in "The glasses I bought from the store downtown that's been open for 20 years and everyone trusts are red," the RNN has likely forgotten "glasses" by the time it reaches "red" — it might confusingly connect "red" to "years" instead. Forgetting, slowness, and poor long-range connections were the ceiling of AI language for years.

The 2015 Band-Aid: The Original Attention Mechanism

Before the Transformer, researchers found a partial fix: Attention. The insight was brilliant in its simplicity — instead of relying on the hidden state to carry everything forward, what if at each step the model could look back at any previous word and focus on the most relevant ones?

This helped significantly. But it was still bolted onto the RNN — it didn't fix the fundamental speed problem, and it added cost on top of an already slow architecture.

2017: The Paper That Changed Everything

Eight researchers at Google Brain — Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin — looked at all these problems and asked the audacious question.

Self-Attention: The Core Innovation

The key mechanism inside the Transformer is Self-Attention. Each word simultaneously asks three questions about every other word:

The attention score for each word pair is computed with one compact formula:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) × V

In plain English: Q·Kᵀ measures how much a query matches a key (compatibility), √dₖ keeps the dot products from getting too large, softmax converts the scores into percentages, and V is the weighted blend of information passed forward. Each word votes on how much attention to pay every other word, and the information from relevant words flows through.

The Dinner Party: Self-Attention as a Conversation

That's self-attention: one ambiguous word, a room full of context, and a weighted blend that resolves the meaning — no formulas needed.

1. Give each word a Q, K, V:
   river       = ([1,0], [1,0], [0.9, 0.1])
   bank        = ([1,1], [1,1], [0.5, 0.5])
   overflowed  = ([0,1], [0,1], [0.1, 0.9])

2. Match bank's Q against every K (compatibility):
   scores = [1.0, 2.0, 1.0], then ÷√2 → [0.71, 1.41, 0.71]

3. Squish into percentages (softmax):
   → [0.26, 0.48, 0.26]   ← the 26/48/26 split

4. Blend the V vectors by those percentages:
   new_bank = [0.50, 0.50]  — now carries river + flood context

For a fuller sentence — "The bank by the river overflowed" — every word scores every other word at once:

Higher score = stronger attention. "bank" scoring 0.92 on "river" is how the model learns this is a riverbank, not a financial institution. (Scores are illustrative; real weights are learned and sum to 1.0 per row after softmax.)

Multi-Head Attention: A Panel of Experts

The dinner-party conversation was only one type of conversation. Real sentences have many relationships at once — grammar, contrast, mood, big-picture meaning — and one conversation can't catch them all. So the Transformer hires a panel of experts, each reading the same sentence through a different lens. Take "The smart glasses are light but their battery is very weak":

Each expert writes its own attention matrix, then the model staples all the reports together into one rich representation. It's the trick a good hospital uses — a cardiologist, neurologist, and radiologist examining the same patient, pooling notes for a diagnosis sharper than any one could produce alone. And the scale is wild: GPT-3 runs 96 of these experts in parallel inside every single layer, and each one learned its niche during training — nobody assigned them.

Positional Encoding: Remembering Order

Reading everything in parallel creates a subtle problem: word order gets lost. Without position info, "The dog bit the man" and "The man bit the dog" look like the same bag of tokens to the attention mechanism.

"dog" + position 1 encoding → knows it's the subject
"bit" + position 2 encoding → knows it's the verb
"man" + position 3 encoding → knows it's the object

Now "The dog(1) bit(2) the man(3)" is mathematically distinct from "The man(1) bit(2) the dog(3)" — order preserved, without losing parallelism.

Inside a Transformer Block

A complete Transformer isn't just attention — it's a stack of blocks, each containing several components, repeated N times:

The Residual Connection is worth calling out: at each layer, the original input is added back to the attention output, so even if a head learns something unhelpful, the original information isn't destroyed. It's the same "add the original input back" trick that let us train deep networks in Part 4 without losing early information.

Every Concept You've Learned Lives Inside This Diagram

The Timeline: From Research to Revolution

RNN vs. Transformer — The Final Scoreboard

The Four Key Components — Summary

Pro Tips for Builders

Try It Yourself

1. Visualize attention. BertViz lets you watch how different attention heads in BERT focus on different words — syntax heads behave differently from semantic heads.
2. Feel the difference. Load bert-base-uncased (encoder) and gpt2 (decoder). BERT sees the whole sentence at once; GPT-2 generates one token at a time. Same architecture, different configurations.

from transformers import pipeline

# BERT (encoder) — sees the full sentence at once and fills the blank
fill_mask = pipeline("fill-mask", model="bert-base-uncased")
print(fill_mask("The bank by the [MASK] overflowed.")[0])
# {'token_str': 'river', 'score': 0.89, ...}
# BERT picks "river" because it reads "overflowed" simultaneously with "bank".

# GPT-2 (decoder) — generates tokens left-to-right
generator = pipeline("text-generation", model="gpt2")
print(generator("The bank by the river", max_new_tokens=5)[0]["generated_text"])
# "The bank by the river was flooded..."

3. Count attention heads.

from transformers import GPT2Config
config = GPT2Config()
print(f"GPT-2 Small: {config.n_head} heads × {config.n_layer} layers "
      f"= {config.n_head * config.n_layer} attention ops")
# GPT-2 Small: 12 heads × 12 layers = 144 attention ops

Key Takeaways

Up Next in the Series