Step by Step
B
Batch Normalization — normalizes across the batch dimension
For each feature, normalizes values across all samples in the current batch. Works great for CNNs, but requires a reasonably large batch size and struggles with variable-length sequences.
Example: in an image classifier, batch normalization computes the mean and standard deviation of a given feature across all images currently in the batch.
L
Layer Normalization — normalizes across the feature dimension
Normalizes across the feature dimension within each individual sample, independent of batch size, and works cleanly with variable-length sequences — making it the standard choice for Transformers and RNNs.
Example: in a Transformer processing a sentence, layer normalization computes statistics across all the features of a single token's representation, regardless of how many other tokens or samples are in the batch.
⚖
Both add learnable scale and shift parameters
After normalizing, both techniques apply learnable scale and shift parameters, allowing the network to recover whatever effective distribution best suits training if pure normalization is too restrictive.
Example: even after normalizing to mean 0 and std 1, a learnable scale and shift parameter can adjust the values as needed for optimal training.
Applied Walkthrough
1
A team is building a Transformer model that processes variable-length text sequences with small batch sizes due to memory constraints.
2
Ask: is Batch Normalization or Layer Normalization the better fit here? Layer Normalization, since it works independently of batch size and handles variable-length sequences cleanly.
3
If the team instead switched to Batch Normalization, they would run into problems with small batches (unreliable statistics) and variable-length sequences (inconsistent normalization axis).
4
This is exactly why Layer Normalization became the standard choice for Transformers and RNNs, while Batch Normalization remains standard for CNNs with large, fixed-size image batches.
Exam Application
Exams test whether you can match the correct normalization technique to the correct architecture: Batch Normalization for CNNs (normalizing across the batch dimension), Layer Normalization for Transformers and RNNs (normalizing across the feature dimension, batch-size independent).
⚠ Common Trap
The most common trap is assuming Batch Normalization is the universal default choice for all architectures, since it's often introduced first. Batch Normalization specifically struggles with small batch sizes and variable-length sequences — exactly the conditions common in Transformers and RNNs — which is why Layer Normalization is the standard there instead.
✓ Quick Self-Check
1. What dimension does Batch Normalization normalize across?
The batch dimension — across all samples in the current batch, for each feature.
Tap to reveal / hide
2. What dimension does Layer Normalization normalize across?
The feature dimension — within each individual sample, independent of batch size.
Tap to reveal / hide
3. Which architectures typically use Batch Normalization?
CNNs.
Tap to reveal / hide
4. Which architectures typically use Layer Normalization?
Transformers and RNNs.
Tap to reveal / hide
5. Why does Layer Normalization work better than Batch Normalization for variable-length sequences?
Because it normalizes within each individual sample rather than across the batch, making it independent of batch size and sequence length inconsistencies.
Tap to reveal / hide