Step by Step
S
Sigmoid — 0 to 1, for binary classification output
Squashes any input into a range between 0 and 1, making it well-suited for binary classification output layers, but it suffers from vanishing gradients when used throughout deep hidden layers.
Example: a binary spam classifier's output layer uses sigmoid to produce a single probability between 0 and 1.
T
Tanh — -1 to 1, zero-centered but still vanishes
Squashes input into a range between -1 and 1, centered at zero (an advantage over sigmoid for hidden layers), but still suffers from vanishing gradients in very deep networks.
Example: tanh might be chosen over sigmoid for a hidden layer specifically because its zero-centered output can help with certain training dynamics, even though both still vanish in very deep networks.
R
ReLU — fastest, avoids vanishing gradient, the standard for hidden layers
Computes max(0, x), making it the fastest to compute and largely avoiding the vanishing gradient problem for positive inputs, which is why it's the standard default choice for hidden layers in deep networks.
Example: a modern deep network's hidden layers almost always default to ReLU rather than sigmoid or tanh, specifically for this speed and vanishing-gradient-avoidance combination.
⊕
Softmax — converts logits to probabilities, for multi-class output
Converts a vector of raw output scores (logits) into a probability distribution across all classes that sums to exactly 1, making it the standard choice for multi-class classification output layers.
Example: a 10-digit classifier's output layer uses softmax to convert 10 raw scores into 10 probabilities that sum to 1, representing the model's confidence in each possible digit.
Applied Walkthrough
1
A team is building a multi-class image classifier that must choose among 5 possible categories.
2
Ask: which activation function belongs at the output layer? Softmax, since it converts raw scores into a proper probability distribution across all 5 classes summing to 1.
3
For the hidden layers throughout the rest of the network, ReLU is the standard choice, given its speed and resistance to vanishing gradients.
4
Sigmoid would be reserved specifically for a binary (2-class) output layer instead, and tanh might occasionally be chosen for certain hidden layers where its zero-centered output offers a specific advantage.
Exam Application
Exams heavily test correctly matching each activation function to its appropriate role: sigmoid for binary output, softmax for multi-class output, and ReLU as the default for hidden layers. This is one of the most frequently and directly tested topics in any neural network course.
⚠ Common Trap
The most common trap is using sigmoid or tanh throughout all hidden layers of a deep network rather than reserving them for specific output-layer roles (sigmoid for binary) — this causes vanishing gradients and stalls training, which is exactly why ReLU became the standard default for hidden layers instead.
✓ Quick Self-Check
1. What range does sigmoid output fall into?
Between 0 and 1.
Tap to reveal / hide
2. What range does tanh output fall into, and what advantage does this offer over sigmoid?
Between -1 and 1, zero-centered, which can help with certain training dynamics in hidden layers compared to sigmoid.
Tap to reveal / hide
3. What makes ReLU the standard default choice for hidden layers?
It's fast to compute and avoids the vanishing gradient problem for positive inputs.
Tap to reveal / hide
4. What does softmax convert logits into, and where is it typically used?
A probability distribution across all classes summing to 1, typically used in multi-class classification output layers.
Tap to reveal / hide
5. Which activation function would you use for a binary classification output layer?
Sigmoid.
Tap to reveal / hide