Step by Step
P
Pretraining — learning from a massive text corpus
A massive Transformer model is trained on a huge text corpus to predict the next token, in a self-supervised manner — the "label" is simply the next token in the text itself, requiring no manual labeling.
Example: training on billions of web pages and books, with the model learning simply by trying to predict what word comes next given the preceding text.
F
Fine-tuning (SFT) — teaching instruction following
Supervised Fine-Tuning trains the pretrained model on a curated dataset of instruction-response pairs, teaching it to actually follow instructions rather than just continuing text in a generic way.
Example: fine-tuning on examples like "Instruction: summarize this article. Response: [good summary]" to teach the model how to actually follow such instructions.
⭐
RLHF — aligning with human values
Reinforcement Learning from Human Feedback has human raters rank multiple model outputs, trains a reward model from those rankings, then uses reinforcement learning to fine-tune the LLM to maximize that reward model's score — aligning the model's behavior with human preferences.
Example: human raters ranking several possible responses to the same prompt from best to worst, with that ranking data used to train a reward model that then guides further RL-based fine-tuning.
✨
Emergent abilities — appearing suddenly at scale
Certain abilities — like few-shot learning, chain-of-thought reasoning, and code generation — appear suddenly once a model reaches a sufficient scale, without being explicitly trained for those specific abilities and without being reliably predictable in advance.
Example: a smaller version of a model might completely fail at a chain-of-thought reasoning task, while a larger version of the same architecture, trained the same way, suddenly performs it well — with no clear signal in advance of exactly when this capability would emerge.
Applied Walkthrough
1
A raw pretrained language model is excellent at predicting plausible next words but doesn't reliably follow instructions or align with what humans actually want.
2
Supervised fine-tuning on curated instruction-response pairs teaches the model to actually follow instructions, rather than just generically continuing text.
3
RLHF then further refines the model's behavior using human preference rankings, aligning its outputs more closely with what humans actually judge as helpful, honest, and harmless.
4
Separately, as models scale up in size during this whole process, certain surprising "emergent" abilities can appear suddenly — capabilities the smaller version of the same architecture simply didn't have, without ever being explicitly trained for that specific ability.
Exam Application
Exams test whether you can correctly sequence and describe the three training stages (pretraining, SFT, RLHF) and whether you understand what makes emergent abilities distinctive — they appear suddenly at scale, are not the product of explicit training for that specific skill, and are not reliably predictable in advance.
⚠ Common Trap
The most common trap is assuming a raw pretrained model (before SFT and RLHF) is already a helpful, aligned assistant. Pretraining alone only teaches the model to predict plausible next tokens — it's the subsequent SFT and RLHF stages that specifically teach instruction-following and alignment with human preferences.
✓ Quick Self-Check
1. What is the training objective during the pretraining stage?
Predicting the next token in a massive text corpus, in a self-supervised manner.
Tap to reveal / hide
2. What does Supervised Fine-Tuning (SFT) teach the model to do?
Follow instructions, by training on curated instruction-response pairs.
Tap to reveal / hide
3. What does RLHF specifically aim to achieve?
Aligning the model's behavior with human preferences, using a reward model trained from human rankings of outputs.
Tap to reveal / hide
4. What are emergent abilities?
Capabilities like few-shot learning or chain-of-thought reasoning that appear suddenly at scale, without being explicitly trained for and without being reliably predictable in advance.
Tap to reveal / hide
5. Is a raw pretrained model (before SFT and RLHF) already a helpful, aligned assistant?
No — pretraining alone only teaches next-token prediction; SFT and RLHF are specifically what teach instruction-following and human alignment.
Tap to reveal / hide