Step by Step
R
The formula — max(0, x)
ReLU (Rectified Linear Unit) computes the maximum of 0 and the input value — an extremely simple calculation compared to sigmoid or tanh.
Example: an input of 7 produces an output of 7 (since 7 is greater than 0); an input of -3 produces an output of 0.
E
Negative inputs — the neuron goes silent
If the input is negative, ReLU outputs exactly zero — the neuron is effectively "silent" and contributes nothing to that forward pass.
Example: like a bouncer at a club door, any negative value is turned away entirely and never lets any of its value through.
L
Positive inputs — pass straight through
If the input is positive, ReLU simply passes the value through completely unchanged — no squashing or scaling at all.
Example: an input of 4.4 passes through as exactly 4.4, unlike sigmoid or tanh which would squash it into a bounded range.
U
Why it's the default — simple, fast, avoids vanishing gradients
ReLU's simplicity makes it very fast to compute, and because it passes positive gradients through unchanged, it avoids the vanishing gradient problem that plagued older activation functions like sigmoid in deep networks.
Example: a 50-layer network using ReLU throughout its hidden layers trains far more reliably than the same network using sigmoid throughout, precisely because of this vanishing gradient advantage.
Applied Walkthrough
1
A hidden layer neuron receives a summed input value of -2.5 during the forward pass.
2
Ask: what does ReLU output for this negative input? Exactly 0 — negative inputs are always zeroed out completely.
3
Contrast: if the summed input had instead been +2.5, ReLU would pass that value through completely unchanged, outputting exactly 2.5.
4
This simple, sharp behavior — zero for negative, unchanged for positive — is exactly why ReLU is fast to compute and avoids the vanishing gradient issues that plague sigmoid and tanh in deep hidden layers.
Exam Application
Exams test the exact formula (max(0, x)) and the two key advantages ReLU offers over sigmoid/tanh in hidden layers: computational speed and avoidance of vanishing gradients for positive inputs. This is a near-guaranteed topic on any neural network exam given how universally ReLU is used as the default hidden-layer activation.
⚠ Common Trap
The most common trap is forgetting that ReLU can still have its own problem — the "dying ReLU" issue, where a neuron that consistently receives negative inputs can get permanently stuck outputting zero, contributing nothing to learning. This is specifically what Leaky ReLU was designed to fix, by allowing a small negative slope instead of a hard zero.
✓ Quick Self-Check
1. What is the formula for ReLU?
max(0, x) — output is 0 for negative inputs, or the input itself unchanged for positive inputs.
Tap to reveal / hide
2. What does ReLU output for a negative input?
Exactly 0.
Tap to reveal / hide
3. What does ReLU output for a positive input?
The input value itself, completely unchanged.
Tap to reveal / hide
4. Why is ReLU generally preferred over sigmoid in hidden layers?
It's computationally faster and avoids the vanishing gradient problem for positive inputs, unlike sigmoid.
Tap to reveal / hide
5. What problem can ReLU still suffer from, and what fixes it?
The dying ReLU problem, where a neuron gets permanently stuck at zero output; Leaky ReLU fixes this by allowing a small negative slope instead of a hard zero.
Tap to reveal / hide