Part 12 — RLHF: How Humans Taught AI to Be Helpful

The answer is RLHF — Reinforcement Learning from Human Feedback. It's not just a training trick. It's the invisible workforce of thousands of human annotators that shaped every AI model you've ever talked to.

The Problem: Intelligence Without Purpose

Before we can understand RLHF, we need to understand what was broken.

In 2020, OpenAI released GPT-3 — at the time, the largest language model ever built. 175 billion parameters. Trained on hundreds of billions of words from the internet, books, and code. It was genuinely extraordinary.

And yet, it was nearly unusable as an assistant.

The technical capability was the same. GPT-3 could have answered the question. It had all the knowledge. The problem was that it was trained to complete text, not to be helpful.

This is a fundamental misalignment between what the model was optimized for (predict the next word accurately) and what users actually needed (understand my intent and respond usefully).

The Three-Stage Pipeline

Modern AI assistants like ChatGPT, Claude, and Gemini are all built using a three-stage training pipeline. Each stage transforms the model in a specific way.

Each stage builds on the previous. You can't skip stages — a model needs language knowledge (Stage 1) before it can learn instruction-following (Stage 2), and it needs basic instruction-following before preference learning (Stage 3) has anything meaningful to optimize.

Stage 1: Pre-Training — Building the Foundation

Pre-training is the most computationally expensive phase. It's also the most passive from a human perspective — no one is writing labels or making decisions. The model simply learns by doing.

After trillions of these predictions across terabytes of text, the model has internalized:

• Grammar and syntax across many languages
• Factual knowledge from encyclopedias, textbooks, news
• Reasoning patterns from scientific papers and code
• Cultural context from books, stories, and forums

The key insight: the model learns all of this without anyone telling it what to learn. The structure of human knowledge is encoded in the statistical patterns of text itself.

What you get after pre-training is called a Base Model. It's extraordinarily capable — but it's a text completion machine, not an assistant. Ask it a question, and it might just generate 3 more related questions.

Stage 2: Supervised Fine-Tuning (SFT) — Teaching Intent

SFT is the bridge between "knows a lot" and "can be useful." It's a relatively small dataset (10,000 to 100,000 examples) compared to pre-training, but it's 100% human-crafted.

The model learns from thousands of such pairs. This teaches it the format of a helpful response: address the question directly, provide structured information, acknowledge trade-offs.

What SFT teaches:

• How to interpret question intent (not just complete text)
• Response formatting (lists, comparisons, steps)
• Appropriate length (not too short, not rambling)
• Basic safety (don't explain how to make weapons)

What SFT can't teach well:

• Subtle quality differences between two decent answers
• Complex safety edge cases
• Stylistic preferences humans have but can't easily articulate

This is where RLHF comes in.

Stage 3: RLHF — The Secret Sauce

RLHF is technically Reinforcement Learning, but the human feedback component makes it fundamentally different from how RL is used in games or robotics. Let's break down the mechanism step by step.

Step 3a: Collect Human Comparisons

Instead of writing ideal answers (hard and slow), human evaluators are shown two (or more) model responses to the same prompt and asked a simple question: Which is better?

Why is comparing easier than writing?

• Faster: Comparing takes 30 seconds vs writing an ideal answer taking 10+ minutes
• More consistent: "A is better than B" is less subjective than writing the perfect answer from scratch
• Captures subtle preferences: Things like tone, confidence level, appropriate caveats — hard to write rules for, easy to recognize

Step 3b: Train the Reward Model

From these millions of human comparisons, a separate neural network is trained — the Reward Model. Its job is simple: given any response, output a single number representing how much humans would like it.

The Reward Model is now a proxy for "what humans want." It can score any response instantly — without requiring a human to read it. This is the key that makes RLHF scale.

Step 3c: Optimize the LLM with PPO

Now we have a "teacher" (the Reward Model) that can score anything. We use this to improve the main LLM using PPO (Proximal Policy Optimization) — a reinforcement learning algorithm.

Why PPO specifically?

PPO is a "safe" reinforcement learning algorithm. It updates the LLM's weights but includes a constraint: don't change too much in one step. Without this constraint, the model would exploit the Reward Model — finding ways to get high scores that don't actually reflect real quality (a phenomenon called "reward hacking").

The PPO constraint keeps the trained model close to the SFT model, ensuring it doesn't forget its language abilities while improving helpfulness.

SFT vs RLHF: What's the Difference?

A helpful analogy:

The key difference: SFT teaches by imitation. RLHF teaches by preference — which is a much richer signal, especially for things that are hard to define explicitly but easy to recognize.

Constitutional AI: The Next Evolution

RLHF requires human evaluators — thousands of them, working constantly. This is expensive and has its own biases (evaluators have different values, fatigue, cultural backgrounds).

Anthropic developed an extension of RLHF called Constitutional AI (CAI), used in Claude:

CAI significantly reduces the human labor required while making the model's values more explicit and auditable. You can see exactly what principles the model was trained to uphold.

Real-World Impact: RLHF Before and After

The measurable improvements from RLHF are dramatic:

The InstructGPT paper (the research behind ChatGPT) showed that humans preferred RLHF-trained outputs over GPT-3 outputs 71% of the time — even when the RLHF model was 100x smaller. RLHF doesn't just improve safety — it makes the model fundamentally more useful.

The Human Workforce Behind RLHF

RLHF requires something we rarely talk about: an enormous amount of human labor.

This human labor is one of the biggest differentiators between frontier AI labs. OpenAI, Anthropic, and Google have invested hundreds of millions of dollars in human annotation infrastructure — and it's a significant competitive moat.

Try It Yourself: Seeing RLHF's Effects

You can explore the difference between base and RLHF models directly through APIs:

from openai import OpenAI

client = OpenAI()

# A prompt that shows the base model vs instruction model difference
ambiguous_prompt = "Baking chocolate cake"

# gpt-3.5-turbo-instruct behaves more like a base/SFT model
# It treats the input as text to complete
base_style_response = client.completions.create(
    model="gpt-3.5-turbo-instruct",
    prompt=ambiguous_prompt,
    max_tokens=150
)
print("Base-style completion:")
print(base_style_response.choices[0].text)
print()

# gpt-4o is RLHF-trained — treats the same input as an implicit question
chat_response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": ambiguous_prompt}]
)
print("RLHF chat model response:")
print(chat_response.choices[0].message.content)

# Demonstrating the "I don't know" behavior (RLHF-trained honesty)
# RLHF models are trained to acknowledge uncertainty

responses_to_test = [
    "What is the population of Mars?",  # Should say "no population" not make up numbers
    "What did Einstein say on March 15, 1945?",  # Specific date — may not know exactly
    "Who won the 2028 Olympics?",  # Future event — should not guess
]

for prompt in responses_to_test:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=100
    )
    print(f"Q: {prompt}")
    print(f"A: {response.choices[0].message.content}")
    print("-" * 40)

# Comparing system prompt effects (a direct RLHF capability)
# Base models can't follow system prompts — only RLHF models can

def test_system_prompt(system_msg, user_msg):
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": system_msg},
            {"role": "user", "content": user_msg}
        ],
        max_tokens=200
    )
    return response.choices[0].message.content

# RLHF-trained models respect role instructions
formal_response = test_system_prompt(
    "You are a formal legal advisor. Respond professionally and conservatively.",
    "What should I do about my neighbor's loud music?"
)

casual_response = test_system_prompt(
    "You are a chill friend. Keep it casual and practical.",
    "What should I do about my neighbor's loud music?"
)

print("Formal (legal advisor):")
print(formal_response)
print("\nCasual (friend):")
print(casual_response)
# Same question, same model — dramatically different responses
# This persona-following is a direct product of RLHF training

Expected output observations:

• Base-style completion will continue the text "Baking chocolate cake" as if writing an article
• RLHF chat model will ask if you want a recipe or offer one directly
• RLHF models will say "I can't know that" for Mars population or 2028 Olympics
• System prompts dramatically change tone and framing in RLHF models

Key Takeaways

What's Next in the Series