Step by Step
0
Zero initialization — total failure via symmetry
If all weights start at zero, every neuron in a layer computes the exact same gradient during backpropagation, meaning symmetry never breaks and the network never learns anything useful.
Example: with zero initialization, 500 neurons in a layer would all behave identically forever, wasting almost all of the network's potential capacity.
↑
Weights too large — activations explode
If randomly initialized weights are too large, activations can grow uncontrollably as they pass through successive layers, leading to unstable training.
Example: overly large initial weights might cause activation values to grow into the millions by just a few layers deep, destabilizing the entire training process.
↓
Weights too small — gradients vanish
If randomly initialized weights are too small, activations (and their corresponding gradients) can shrink toward zero as they pass through successive layers, contributing to the vanishing gradient problem.
Example: overly small initial weights compound the vanishing gradient issue right from the very start of training, before any learning has even had a chance to begin.
✓
Match the initializer to the activation function
Xavier (Glorot) initialization is designed for sigmoid/tanh; He initialization is designed for ReLU. PyTorch applies He initialization by default for linear and convolutional layers, reflecting how standard ReLU has become.
Example: choosing He initialization for a network using ReLU throughout its hidden layers, rather than Xavier, which was designed with different activation characteristics in mind.
Applied Walkthrough
1
A student initializes a deep ReLU network using Xavier initialization instead of He initialization, assuming any "proper" random initialization scheme would work equally well.
2
Ask: does this mismatch matter? Yes — Xavier was specifically designed for sigmoid/tanh's characteristics, not ReLU's, since ReLU zeros out roughly half of all neuron outputs.
3
Using the mismatched initializer can lead to activations that are poorly scaled for ReLU's specific behavior, hurting early training dynamics even before any weight updates have occurred.
4
The fix: use He initialization specifically for ReLU-based networks, matching the initializer to the activation function actually being used, exactly as PyTorch does by default for linear and conv layers.
Exam Application
Exams test whether you understand that initialization isn't a minor implementation detail but a genuinely critical hyperparameter — with zero initialization being catastrophic, and Xavier vs. He needing to be correctly matched to the activation function in use (sigmoid/tanh vs. ReLU respectively).
⚠ Common Trap
The most common trap is assuming any reasonable random initialization scheme is interchangeable, or assuming a network can "recover" from a poor initialization choice given enough training time. In practice, a poor initialization choice can meaningfully hurt training dynamics from the very start, which is why matching the initializer to the activation function matters.
✓ Quick Self-Check
1. What happens if all weights are initialized to zero?
Every neuron computes identical gradients during backpropagation, so symmetry never breaks and the network never learns anything useful.
Tap to reveal / hide
2. What happens if initial weights are too large?
Activations can explode, growing uncontrollably as they pass through successive layers.
Tap to reveal / hide
3. What happens if initial weights are too small?
Gradients can vanish, shrinking toward zero as they pass through successive layers.
Tap to reveal / hide
4. Which initialization scheme is designed for ReLU, and which is designed for sigmoid/tanh?
He initialization is designed for ReLU; Xavier (Glorot) initialization is designed for sigmoid/tanh.
Tap to reveal / hide
5. What does PyTorch use as its default initialization for linear and convolutional layers?
He initialization.
Tap to reveal / hide