Step by Step
1
The problem — base LLMs are knowledgeable but unhelpful
Base LLMs, pretrained purely on internet text, are knowledgeable but fundamentally unhelpful as assistants — they're built to complete text in a plausible way, not to follow instructions or align with what a user actually wants.
Example: a raw base LLM given an instruction might simply continue the text in a generic way, rather than actually recognizing and following the instruction as intended.
2
Collect human feedback — which response is better?
Human raters compare multiple candidate model outputs for the same prompt and indicate which one they prefer.
Example: showing human raters two different possible responses to the same prompt and asking them to indicate which one is more helpful, honest, or appropriate.
3
Train a reward model — predicting human preferences
A separate reward model is trained on these human preference comparisons, learning to predict which kinds of outputs humans are likely to prefer.
Example: training a reward model that can score any given response with a number representing how much a human rater would likely prefer it.
4
Fine-tune the LLM with RL — maximizing the reward
Reinforcement learning (typically PPO) is then used to fine-tune the LLM to maximize the reward model's predicted score, producing a model that follows instructions, refuses harmful requests, and gives genuinely helpful responses.
Example: ChatGPT, Claude, and Gemini all use variants of this RLHF process to transform a raw, knowledgeable-but-unhelpful base model into a genuinely useful, aligned assistant.
Applied Walkthrough
1
A raw, pretrained base LLM is knowledgeable about a huge range of topics but tends to just continue text in generic ways rather than actually following user instructions.
2
Human raters are shown multiple candidate responses to various prompts and indicate which response they prefer in each case.
3
This human preference data is used to train a separate reward model, which learns to predict what kinds of responses humans are likely to prefer.
4
Reinforcement learning (typically PPO) then fine-tunes the original LLM to maximize this reward model's score, transforming it into a model that reliably follows instructions, refuses harmful requests, and gives genuinely helpful responses — exactly the transformation RLHF is responsible for in systems like ChatGPT, Claude, and Gemini.
Exam Application
Exams test whether you can correctly sequence RLHF's three stages (collect human feedback, train a reward model, fine-tune the LLM via RL to maximize that reward) and whether you understand WHY this process is necessary — base pretrained LLMs are knowledgeable but not inherently helpful or aligned with human preferences.
⚠ Common Trap
The most common trap is assuming the reward model and the final LLM are the same model. They are two separate models working together in sequence — the reward model is trained first from human preference data, and then used to guide the separate reinforcement learning fine-tuning process applied to the actual LLM.
✓ Quick Self-Check
1. Why are base pretrained LLMs considered "unhelpful" before RLHF?
Because they're built to complete text plausibly, not to follow instructions or align with what a user actually wants.
Tap to reveal / hide
2. What do human raters do in the first stage of RLHF?
Compare multiple candidate model outputs for the same prompt and indicate which one they prefer.
Tap to reveal / hide
3. What does the reward model learn to do?
Predict which kinds of outputs humans are likely to prefer, based on the collected human preference data.
Tap to reveal / hide
4. What reinforcement learning algorithm is typically used to fine-tune the LLM in the final RLHF stage?
PPO (Proximal Policy Optimization).
Tap to reveal / hide
5. Are the reward model and the final fine-tuned LLM the same model?
No — they are two separate models; the reward model guides the reinforcement learning process applied to the separate LLM.
Tap to reveal / hide