Step by Step
W
Weights — how much each input matters
Each input to a neuron is multiplied by a learned weight, representing how much that particular input should influence the neuron's output.
Example: in a neuron with two inputs, weight w1 might be large (that input matters a lot) while weight w2 might be near zero (that input barely matters).
I
Input — the raw values feeding the neuron
The actual numeric values coming into the neuron, whether from the original data or from the outputs of a previous layer's neurons.
Example: pixel brightness values feeding into the first layer of an image classifier.
S
Summation — combining weighted inputs plus a bias
All the weighted inputs are added together along with a bias term, producing a single combined value: z = w1x1 + w2x2 + b.
Example: z = (0.8 × 3) + (0.2 × 5) + 1 = 4.4, combining two weighted inputs and a bias into one number.
E
Excitation (Activation) — introducing nonlinearity
The summed value passes through an activation function, introducing nonlinearity. Without this step, stacking any number of layers would still mathematically collapse into just one linear function.
Example: applying ReLU to the summed value of 4.4 simply passes it through unchanged (since it's positive), but a negative summed value would be zeroed out entirely.
Applied Walkthrough
1
A student builds a 50-layer network but forgets to include any activation functions between layers.
2
Ask: does adding more layers without activation functions actually increase the network's power? No — without nonlinear activations, any number of stacked linear layers mathematically collapses into a single linear function.
3
This means the 50-layer network, despite its depth, can only learn simple linear relationships — no better than a single-layer linear model.
4
The fix: insert a nonlinear activation function (like ReLU) after each layer's summation step, which is what actually gives depth its power, per the Universal Approximation Theorem.
Exam Application
Exams frequently test whether you understand WHY activation functions are essential, not optional — the mathematical fact that stacking purely linear operations always collapses into one linear function is a foundational concept tested across nearly every neural network course.
⚠ Common Trap
The most common trap is assuming that simply adding more layers automatically makes a network more powerful, regardless of whether activation functions are present. Depth without nonlinearity provides zero additional expressive power over a single linear layer — the activation function is what actually unlocks the benefit of depth.
✓ Quick Self-Check
1. What does the 'W' in WISE represent?
Weights — the learned values that determine how much each input influences the neuron's output.
Tap to reveal / hide
2. What formula represents the summation step in a neuron?
z = w1x1 + w2x2 + b — weighted inputs summed together plus a bias term.
Tap to reveal / hide
3. What happens if you stack many layers without any activation functions between them?
The entire stack mathematically collapses into just one linear function, no matter how many layers are stacked.
Tap to reveal / hide
4. What does the Universal Approximation Theorem state?
That a network with nonlinear activations can, in principle, approximate any continuous function.
Tap to reveal / hide
5. What is the role of the activation function in a neuron?
Introducing nonlinearity, which is what actually gives depth its expressive power in deep learning.
Tap to reveal / hide