Part 5 — How AI Actually Learns: The Training Loop Explained

In the previous article we built an artificial neuron and learned that it has weights — importance multipliers that determine how much each input influences the output. The question we left open: how does the AI learn the right weights?

The answer is the Training Loop — four steps, repeated millions of times, that turn random numbers into intelligence.

The Core Idea: Learning from Mistakes

Think about how a child learns to walk. Nobody programs the angles their legs need to maintain. Nobody writes rules for balance. The child:

1. Tries to take a step
2. Falls over
3. Somehow figures out what went wrong
4. Adjusts the next attempt
5. Repeats — until walking becomes automatic

An AI learns exactly the same way. The only difference is speed: a neural network can "fall" and "adjust" millions of times in a few hours.

The Training Loop: 4 Steps

Let's go through each step with a concrete example: classifying whether a device is smart glasses or a smart ring.

Step 1: Forward Pass — Make a Guess

Data enters the network at the input layer and flows forward through every neuron until it produces an output. We call this the Forward Pass.

At the very start of training, all the weights are random. So the output is essentially a random guess.

The network isn't "bad" for being wrong here. It starts wrong. The whole point of training is to make it less wrong, step by step.

Step 2: Loss — Measure the Mistake

"How wrong was the guess?" is the job of the Loss Function (also called the Cost Function).

The most common version is Mean Squared Error (MSE):

loss = (true_label - prediction) ** 2

If we predicted 60% glasses and the true answer is 100% glasses:

loss = (1.0 - 0.60) ** 2 = 0.40 ** 2 = 0.16

A loss of 0.16 on a scale of 0–1. High is bad. Zero is perfect.

Note: For multi-class problems (3+ categories), Cross-Entropy loss is more common than MSE — it handles probability distributions better and trains faster on classification tasks.

Step 3: Backpropagation — Find Who's Responsible

This is the magic step. Once we know the total loss, we need to figure out: which weights caused the error, and by how much?

Imagine a factory with 1,000 workers. The product came out defective. How do you fix it?

Backpropagation is the mathematical version of that second approach. It uses calculus (specifically the chain rule) to calculate the exact contribution of each weight to the total loss.

Think of it like tracing a string of Christmas lights: one bulb goes out and the whole string fails. You don't replace every bulb — you trace backwards from the dead end of the string to find which single bulb broke the chain. Backpropagation does this mathematically, tracing backwards from the output error through every layer to find which weights contributed most.

The output: a number for each weight called the gradient — which tells us: "if we increase this weight by a tiny amount, how much does the loss increase or decrease?"

Step 4: Weight Update — Fix a Little Bit (Gradient Descent)

Now we know which way to adjust each weight. But how much should we adjust?

Too little: training takes forever. Too much: the network overshoots and bounces around without ever converging.

The formula for updating each weight is:

new_weight = old_weight - (learning_rate × gradient)

The Learning Rate is the key hyperparameter here. Think of it as the size of each step when walking down a hill toward the lowest point (minimum loss):

This process of adjusting weights following the gradient is called Gradient Descent — mathematically walking downhill on the loss landscape.

Key Training Vocabulary

Three terms appear in every AI paper and framework:

The Problem That Derails Training: Overfitting

Here's the trap: a network can get very good at the training data while becoming terrible at real-world data it's never seen. This is called Overfitting — the AI memorized the answers instead of learning the pattern.

This is exactly why the embedding model from Article 2 needed to train on billions of multilingual sentence pairs — a smaller dataset would have overfit to memorized phrases rather than learning the underlying geometry of meaning.

Four Ways to Fix Overfitting

1. More Data — The most reliable fix. If the network has seen 100,000 examples instead of 100, memorizing becomes impossible. It has to generalize.

2. Dropout — During training, randomly "turn off" some neurons in each forward pass. The network is forced to not rely on any single neuron, so it develops redundant, distributed knowledge.

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10, 64),
    nn.ReLU(),
    nn.Dropout(p=0.3),   # 30% of neurons randomly disabled during training
    nn.Linear(64, 3),
    nn.Softmax(dim=1)
)

3. Early Stopping — Monitor validation loss (on data the network hasn't trained on). When validation loss starts rising while training loss keeps falling — stop. The network has started memorizing.

4. Data Augmentation — For images: flip, rotate, change brightness, add noise. For text: paraphrase, translate and back-translate. The network sees the same concept presented differently, so it learns the concept — not the presentation.

Complete Python Implementation

Here's the full training loop working end-to-end to classify devices as glasses vs. rings:

import numpy as np
# This is a single-neuron version of the neuron we built in the previous article

# Training data: [price_normalized, weight_normalized] → label
X_train = np.array([
    [0.55, 0.48],   # mid price, mid weight → glasses
    [0.45, 0.72],   # lower price, heavier  → glasses
    [0.35, 0.03],   # low price, very light → ring
    [0.20, 0.03],   # very low, very light  → ring
])
y_train = np.array([1, 1, 0, 0])   # 1=glasses, 0=ring

# Initial weights (random start)
weights       = np.array([0.5, 0.5])
bias          = 0.0
learning_rate = 0.1

# The training loop
for epoch in range(100):
    total_loss = 0

    for x, y_true in zip(X_train, y_train):
        # Step 1: Forward Pass — make a prediction
        prediction = np.clip(np.dot(x, weights) + bias, 0, 1)

        # Step 2: Loss — measure the mistake
        loss        = (y_true - prediction) ** 2
        total_loss += loss

        # Step 3 + 4: Backprop + Weight Update
        error    = y_true - prediction
        weights += learning_rate * error * x
        bias    += learning_rate * error

    if epoch % 25 == 0:
        print(f"Epoch {epoch:3d}  Loss={total_loss:.4f}  "
              f"Weights=[{weights[0]:.3f}, {weights[1]:.3f}]")

Output:

Epoch   0  Loss=0.4823  Weights=[0.618, 0.523]
Epoch  25  Loss=0.1204  Weights=[0.743, 0.611]
Epoch  50  Loss=0.0312  Weights=[0.819, 0.684]
Epoch  75  Loss=0.0089  Weights=[0.867, 0.731]
Epoch  99  Loss=0.0021  Weights=[0.891, 0.752]

The loss dropped from 0.48 to 0.002 in 100 epochs. Now test on a new device:

test = np.array([0.50, 0.55])   # new device: mid price, mid weight
pred = np.clip(np.dot(test, weights) + bias, 0, 1)
label = "Glasses ✅" if pred > 0.5 else "Ring ❌"
print(f"Prediction: {pred:.2f} → {label}")
# Prediction: 0.98 → Glasses ✅

The network learned to distinguish glasses from rings — without a single rule written explicitly. It learned the pattern from 4 examples, 100 epochs, and the four-step training loop.

Real-World Scale

The base model behind early ChatGPT (GPT-3) was trained on roughly 300 billion tokens of text (about 500 billion words — most of the internet). The training loop ran for months on thousands of GPUs running in parallel. The estimated compute cost: over $100 million.

Our example: 4 examples, 100 epochs, 0.001 seconds.

The math is identical. The scale is incomprehensible.

How This Loop Created the 384-Dimensional Embeddings from Article 2

In Article 2, we used a model that converted any sentence into a 384-dimensional vector. Now you know exactly how that model was built:

The Core Insight

Every AI capability you've ever used — image recognition, translation, text generation, code completion — is the result of this loop running billions of times on massive amounts of data.

Try It Yourself

Experiment with the learning rate in the code above:

# Experiment 1: learning_rate = 0.9  (too large)
# Change the learning_rate line to 0.9 and re-run.
# Watch the loss BOUNCE — it overshoots the minimum and never converges.

# Experiment 2: learning_rate = 0.001  (too small)
# Loss drops but very slowly — training would need 10x more epochs.

# Experiment 3: learning_rate = 0.1   (just right — default above)
# Smooth, steady convergence. Loss reaches near-zero by epoch 99.

Try adding a 5th training example that contradicts the pattern slightly — watch how the loss floor rises. That's the model struggling to generalize. This is overfitting in miniature.