Step by Step
B
Batch Gradient Descent — stable but slow
Computes the gradient using the entire dataset before making a single weight update, producing very stable but computationally slow updates, especially for large datasets.
Example: computing the gradient across all 1 million training examples before updating the weights even once.
S
SGD (Stochastic Gradient Descent) — fast but very noisy
Computes the gradient using just one single training example at a time, making updates very fast but also very noisy and erratic step to step.
Example: updating the weights after processing just one training example, then immediately updating again after the very next example.
M
Mini-Batch — the standard, best of both worlds
Computes the gradient using a small batch of examples (typically 32-256), balancing the stability of full batch gradient descent with the speed of SGD — this is the standard approach in deep learning today.
Example: updating weights after processing a mini-batch of 64 examples, rather than either a single example or the entire dataset.
A
Adam — adaptive per-parameter learning rate, the most popular optimizer
Combines momentum (inertia that helps escape saddle points and avoid oscillation) with an adaptive per-parameter learning rate, making it the most popular default optimizer, typically working well with default settings like lr=0.001.
Example: using Adam with its default learning rate of 0.001 as a reasonable starting point for most new deep learning projects.
Applied Walkthrough
1
A team training a deep learning model on a huge dataset needs to balance training stability against computational speed.
2
Ask: should they use full batch gradient descent, SGD, or mini-batch? Mini-batch, since it's specifically designed to balance the stability of full-batch and the speed of SGD.
3
They further decide to use the Adam optimizer on top of mini-batch updates, since Adam's adaptive per-parameter learning rate and momentum help navigate the loss landscape more effectively than plain gradient descent.
4
If training becomes unstable (loss oscillating or diverging), the learning rate — considered the single most critical hyperparameter — is the first thing worth adjusting, typically by lowering it.
Exam Application
Exams test whether you can distinguish the tradeoffs between batch, SGD, and mini-batch gradient descent, and whether you understand why the learning rate is considered the single most critical hyperparameter: too high causes divergence, too low causes painfully slow convergence.
⚠ Common Trap
The most common trap is assuming SGD's noisiness is purely a downside. In practice, the noise in SGD (and mini-batch) updates can actually help the optimizer escape shallow local minima or saddle points that a smoother, full-batch update might get stuck in — noise isn't purely negative.
✓ Quick Self-Check
1. What is the main downside of Batch Gradient Descent?
It's slow, since it requires computing the gradient across the entire dataset before each single weight update.
Tap to reveal / hide
2. What is the main downside of SGD (Stochastic Gradient Descent)?
It's very noisy, since each update is based on just one training example.
Tap to reveal / hide
3. What is the typical range of examples used in a mini-batch?
32 to 256 examples.
Tap to reveal / hide
4. What two things does the Adam optimizer combine?
Momentum (inertia) and an adaptive per-parameter learning rate.
Tap to reveal / hide
5. Why is the learning rate considered the single most critical hyperparameter?
Because setting it too high causes training to diverge, while setting it too low causes training to converge extremely slowly.
Tap to reveal / hide