Step by Step
Z
Zero initialization — a catastrophic symmetry problem
If all weights start at exactly zero, every neuron in a layer computes the exact same output and receives the exact same gradient during backpropagation, meaning the network never diversifies — all neurons remain identical forever.
Example: with zero initialization, 100 neurons in a hidden layer would all behave identically throughout training, effectively acting as if there were only 1 neuron instead of 100.
X
Xavier initialization — designed for sigmoid and tanh
Scales initial weights by 1/sqrt(n_in), where n_in is the number of inputs to a layer, designed specifically to work well with sigmoid and tanh activation functions.
Example: a layer using tanh activations would typically use Xavier initialization to keep the initial signal magnitudes appropriately scaled.
H
He initialization — designed for ReLU
Scales initial weights by 2/sqrt(n_in), accounting for the fact that ReLU zeros out roughly half of all neuron outputs, requiring a different scaling factor than Xavier.
Example: a layer using ReLU activations would typically use He initialization rather than Xavier, since ReLU's characteristics require a different scaling constant.
Applied Walkthrough
1
A student initializes all of a network's weights to exactly zero, thinking it's a simple, neutral starting point.
2
After training, every neuron within each layer is found to have learned the exact same thing, contributing no diversity at all.
3
This is the symmetry problem: with zero initialization, every neuron receives the exact same gradient during backpropagation and updates identically, forever.
4
The fix: use random initialization (Xavier for sigmoid/tanh layers, He for ReLU layers), which breaks this symmetry by giving each neuron a different starting point.
Exam Application
Exams test whether you understand WHY zero initialization catastrophically fails (the symmetry problem) and whether you can correctly match Xavier initialization to sigmoid/tanh layers versus He initialization to ReLU layers. Also expect a note that batch normalization reduces a network's sensitivity to the specific initialization scheme chosen.
⚠ Common Trap
The most common trap is assuming initialization doesn't matter much, or that zero initialization is a safe, neutral default. Zero initialization is catastrophic specifically because it prevents neurons from ever diversifying — this is one of the most fundamental and heavily tested facts about training deep networks.
✓ Quick Self-Check
1. Why does zero initialization fail catastrophically?
Because all neurons receive identical gradients during backpropagation and update identically, so the network never diversifies — the symmetry problem.
Tap to reveal / hide
2. Which initialization scheme is designed for sigmoid and tanh activations?
Xavier initialization.
Tap to reveal / hide
3. Which initialization scheme is designed for ReLU activations?
He initialization.
Tap to reveal / hide
4. Why does He initialization use a different scaling factor than Xavier?
Because ReLU zeros out roughly half of all neuron outputs, requiring a different scaling constant (2/sqrt(n_in) instead of 1/sqrt(n_in)).
Tap to reveal / hide
5. What technique reduces a network's sensitivity to the specific weight initialization scheme chosen?
Batch Normalization.
Tap to reveal / hide