Step by Step
D
During training — randomly disable neurons
On each training pass, a percentage of neurons (typically 20-50%) are randomly turned off entirely, meaning they contribute nothing to that particular forward and backward pass.
Example: with 40% dropout, roughly 4 out of every 10 neurons in a layer are randomly disabled on any given training step, and a different random 4 might be disabled on the next step.
R
Redundant representations — no single neuron can be relied on
Because different neurons get randomly disabled each time, the network is forced to learn redundant, distributed representations rather than concentrating critical information in any single neuron.
Example: like a sports team practicing with random star players benched each session, forcing every player to develop real skill rather than the team relying entirely on one or two stars.
O
Only during training — all neurons active at test time
Dropout is applied only during training. At test/inference time, all neurons are active and outputs are appropriately scaled to compensate for the neurons that were being randomly dropped during training.
Example: a model trained with 30% dropout uses all its neurons fully active when making real predictions after training is complete.
Applied Walkthrough
1
A network trained without any dropout achieves 99% training accuracy but only 65% validation accuracy — a clear overfitting signature.
2
The team adds 40% dropout to the hidden layers and retrains from scratch.
3
During each training step, roughly 40% of neurons are randomly disabled, forcing the remaining neurons to compensate and learn more robust, less narrowly specialized representations.
4
After retraining with dropout, the training/validation accuracy gap shrinks significantly, since the network can no longer overfit by over-relying on a small number of highly specialized neurons.
Exam Application
Exams test whether you understand dropout is applied ONLY during training, with all neurons active during inference/test time, and whether you know the typical dropout rate range (20-50%). Also expect questions asking you to identify dropout as a fix for a described overfitting scenario.
⚠ Common Trap
The most common trap is assuming dropout stays active during inference/testing the same way it does during training. Dropout is exclusively a training-time technique — at test time, the full network with all neurons active is used, with appropriate scaling applied to account for the neurons that were dropped during training.
✓ Quick Self-Check
1. What percentage of neurons does dropout typically disable during a training step?
20-50%, chosen randomly each pass.
Tap to reveal / hide
2. Why does forcing random neurons to disable prevent overfitting?
It forces the network to learn redundant, distributed representations rather than relying too heavily on any single neuron.
Tap to reveal / hide
3. Is dropout active during training, testing, or both?
Only during training — at test/inference time, all neurons are active.
Tap to reveal / hide
4. What analogy is used to explain why dropout works?
Randomly benching star players in practice so the whole team develops real skill, rather than the team relying entirely on a few standout players.
Tap to reveal / hide
5. What kind of problem is dropout specifically designed to address?
Overfitting.
Tap to reveal / hide