Step by Step
1
SGD — simple, but often oscillates
Stochastic Gradient Descent updates weights directly using the current gradient, with no memory of past updates, which can cause oscillation, especially in narrow valleys of the loss landscape.
Example: plain SGD navigating a narrow, curved valley in the loss landscape might zigzag back and forth rather than moving smoothly toward the minimum.
2
Momentum — adds inertia to accelerate consistent directions
Builds on SGD by adding a form of "inertia" — accumulating a moving average of past gradients so the optimizer accelerates in directions it's been consistently moving, smoothing out oscillation.
Example: like a ball rolling downhill picking up speed in a consistent direction, Momentum helps the optimizer push through small bumps and oscillations that would slow down plain SGD.
3
RMSProp — adaptive per-parameter learning rate
Adjusts the effective learning rate separately for each parameter based on the recent magnitude of its gradients, helping parameters with small, infrequent gradients still make meaningful progress.
Example: a rarely-updated parameter with historically small gradients gets an effectively larger learning rate boost under RMSProp compared to frequently-updated parameters.
4
Adam & AdamW — combining momentum and adaptive rates, plus proper weight decay
Adam combines Momentum and RMSProp, becoming the most popular default optimizer (typical defaults: lr=0.001, beta1=0.9, beta2=0.999). AdamW fixes a subtle flaw in how Adam originally implemented weight decay, leading to better generalization — making it the standard choice for fine-tuning large language models.
Example: fine-tuning a pretrained Transformer model today would almost always use AdamW rather than plain Adam, specifically because of its improved weight decay handling.
Applied Walkthrough
1
A team is fine-tuning a large pretrained language model and needs to choose an optimizer.
2
Ask: should they use plain Adam or AdamW? AdamW, since it's the standard choice specifically for fine-tuning large language models due to its properly implemented weight decay.
3
Using plain Adam instead could lead to slightly worse generalization, since Adam's original weight decay implementation had a subtle flaw that AdamW specifically fixed.
4
This progression — SGD to Momentum to RMSProp to Adam to AdamW — shows each optimizer fixing a specific limitation of the one before it.
Exam Application
Exams test whether you understand the historical progression (each optimizer solving a specific problem with its predecessor) and whether you know AdamW's specific improvement over Adam (proper weight decay) and why that makes it the standard for fine-tuning large language models today.
⚠ Common Trap
The most common trap is assuming Adam and AdamW are interchangeable or that the difference between them is trivial. The weight decay fix in AdamW is specifically what makes it the preferred choice for fine-tuning large pretrained models — using plain Adam instead is a commonly tested distinction.
✓ Quick Self-Check
1. What problem does Momentum fix compared to plain SGD?
It reduces oscillation by adding inertia, accelerating movement in directions the optimizer has been consistently moving.
Tap to reveal / hide
2. What does RMSProp adjust that plain SGD and Momentum do not?
It adapts the learning rate separately for each parameter based on the recent magnitude of its gradients.
Tap to reveal / hide
3. What two techniques does Adam combine?
Momentum and RMSProp.
Tap to reveal / hide
4. What does AdamW fix compared to plain Adam?
AdamW implements weight decay properly, leading to better generalization.
Tap to reveal / hide
5. What is AdamW specifically considered the standard choice for?
Fine-tuning large language models.
Tap to reveal / hide