Step by Step
E
Epochs — how many complete passes through the training set
One epoch means the model has seen every training example exactly once. More epochs generally mean more learning, but too many risk overfitting as the model starts memorizing training data.
Example: training for 50 epochs means the model has processed the entire training dataset 50 separate times, in full.
B
Batch size — how many samples per weight update
Determines how many training examples are processed together before the model's weights are updated once. Small batches (8-32) give noisier updates but tend to generalize better; large batches (256-1024) train faster and more smoothly but may overfit more easily.
Example: a batch size of 16 produces noisier, more varied gradient estimates step to step, compared to a batch size of 512 which produces smoother, more consistent gradient estimates.
L
Learning rate — how big each weight update step is
Considered the single most important hyperparameter. Too high causes training to diverge (never settling down); too low causes training to converge extremely slowly or not at all in a reasonable time. A learning rate finder tool, or simply starting at 1e-3 with Adam, are both reasonable starting approaches.
Example: a learning rate set far too high might cause the loss value to increase wildly instead of decreasing, a clear sign of divergence.
Applied Walkthrough
1
A model trains for 200 epochs with a very large batch size of 1024 and a learning rate that turns out to be far too high.
2
The loss value oscillates wildly and never settles down, no matter how many epochs pass.
3
Ask: which of the three dials is most likely the root problem here? The learning rate, since a value that's too high causes divergence regardless of how many epochs or what batch size are used.
4
Reducing the learning rate first, rather than adjusting epochs or batch size, is the standard troubleshooting priority, given that learning rate is considered the single most important hyperparameter.
Exam Application
Exams test whether you can distinguish the distinct roles of these three hyperparameters and correctly diagnose training problems: a model that never settles (likely too-high learning rate), a small-batch model that generalizes well but trains slowly (batch size tradeoff), or a model whose accuracy keeps improving with more full passes through data (epochs).
⚠ Common Trap
The most common trap is assuming these three hyperparameters can compensate for each other equally, or assuming the epoch count is the most important lever when performance is poor. Learning rate is specifically singled out as the single most important hyperparameter — usually the first place to investigate when training doesn't go well.
✓ Quick Self-Check
1. What does one epoch represent?
One complete pass through the entire training dataset.
Tap to reveal / hide
2. What tradeoff exists between small and large batch sizes?
Small batches (8-32) give noisier updates but tend to generalize better; large batches (256-1024) train faster and more smoothly but may overfit more easily.
Tap to reveal / hide
3. What happens if the learning rate is set too high?
Training diverges — the loss oscillates wildly or increases instead of decreasing.
Tap to reveal / hide
4. What happens if the learning rate is set too low?
Training converges extremely slowly, potentially never reaching a good solution in a reasonable amount of time.
Tap to reveal / hide
5. Which of the three hyperparameters (epochs, batch size, learning rate) is considered the single most important?
Learning rate.
Tap to reveal / hide