Step by Step
1
Error signals flow backward, layer by layer
During backpropagation, the error signal (gradient) flows backward through each layer of the network, one at a time, being multiplied by a small number at each step.
Example: at each layer, the gradient might get multiplied by a factor around 0.1, gradually shrinking as it moves backward.
2
Small multiplications compound exponentially
If each layer multiplies the gradient by a small number like 0.1, after just 10 layers the cumulative effect is 0.1 raised to the 10th power — an almost imperceptibly tiny number.
Example: 0.1^10 = 0.0000000001 — by the time this signal reaches the earliest layers, it's essentially indistinguishable from zero.
3
Early layers effectively stop learning
By the time the nearly-zero gradient signal reaches the earliest layers of a deep network, those layers receive essentially no meaningful update signal at all, and effectively stop learning entirely.
Example: in a poorly-designed 20-layer network using sigmoid throughout, the first several layers might barely change at all across the entire training process.
Applied Walkthrough
1
A very deep network is trained using sigmoid activations throughout every layer, just like older architectures from decades ago.
2
Ask: using the whisper-down-the-lane image, what happens to the gradient signal as it travels backward through many layers? It fades exponentially, becoming nearly imperceptible by the time it reaches the earliest layers.
3
This means the earliest layers of the network essentially stop learning, since they receive almost no meaningful gradient signal to update from.
4
This exact problem crippled deep network training for years, until modern solutions — ReLU activations and residual (skip) connections — were specifically developed to keep gradients "alive" as they travel backward through many layers.
Exam Application
Exams test whether you understand the exponential math behind why vanishing gradients occur (small per-layer multipliers compounding across many layers) and whether you can name the modern fixes — ReLU activations and residual/skip connections — that specifically solved this problem and enabled much deeper networks to train successfully.
⚠ Common Trap
The most common trap is thinking the vanishing gradient problem is just a minor inconvenience rather than a fundamental barrier that historically prevented deep networks (10+ layers) from training at all. This problem is precisely why architectures deeper than a few layers were largely impractical before ReLU and residual connections were developed.
✓ Quick Self-Check
1. What happens to the gradient signal as it flows backward through many layers with small per-layer multipliers?
It shrinks exponentially, becoming nearly zero by the time it reaches the earliest layers.
Tap to reveal / hide
2. Using the example of a 0.1 multiplier per layer, what is the cumulative effect after 10 layers?
0.1^10 = 0.0000000001 — an almost imperceptibly tiny gradient.
Tap to reveal / hide
3. What happens to early layers when they receive an essentially zero gradient?
They effectively stop learning, since they receive no meaningful signal to update their weights from.
Tap to reveal / hide
4. Name two modern innovations that specifically solved the vanishing gradient problem.
ReLU activation functions and residual (skip) connections.
Tap to reveal / hide
5. What kind of networks were historically hardest to train because of vanishing gradients?
Very deep networks (many layers), especially those using sigmoid or tanh activations throughout.
Tap to reveal / hide