Step by Step
1
Forward pass — data flows forward, producing a prediction
Input data flows through the network layer by layer, each performing its weighted-sum-and-activation calculation, ultimately producing a final prediction.
Example: an image flows through all layers and the network outputs "78% confident this is a cat."
2
Loss — measuring the regret
A loss function compares the prediction to the true answer, producing a single number representing how wrong (how much "regret") the prediction was.
Example: if the true answer was "dog," the loss function produces a high error value reflecting the network's incorrect confident guess.
3
Backward pass — regret flows backward via the chain rule
The error flows BACKWARD through the network, using the chain rule of calculus to calculate exactly how much each individual weight contributed to that error — assigning mathematical "blame" to each weight.
Example: backpropagation determines a specific weight in an early layer contributed only slightly to the error, while a weight in a later layer contributed heavily.
4
Weight update — adjusting based on assigned blame
Each weight is updated using the formula w = w - alpha × gradient, nudging it in the direction that reduces the loss. The Adam optimizer, with an adaptive per-parameter learning rate, is the standard default (typically starting at lr=0.001).
Example: a weight found to have contributed heavily to the error gets adjusted more substantially than a weight that barely contributed.
Applied Walkthrough
1
A network makes a confidently wrong prediction, generating a high loss value — a large amount of "regret."
2
That regret doesn't just disappear — it flows backward through every layer of the network via the chain rule of calculus.
3
As it flows backward, each individual weight throughout the network gets assigned its own specific share of blame for the error, proportional to how much it contributed.
4
Finally, each weight is updated (nudged) in the direction that would have reduced the error, using the Adam optimizer's adaptive learning rate — and the entire cycle repeats with the next batch of data.
Exam Application
Exams commonly ask you to place the backpropagation steps in the correct order (forward pass, loss, backward pass, weight update) and to distinguish between what backpropagation itself calculates (the gradients — the "blame") versus what the optimizer actually does with those gradients (the weight update itself).
⚠ Common Trap
The most common trap is treating backpropagation and the weight update as the same single step. Backpropagation only calculates the gradients (how much blame each weight deserves) using the chain rule — it is the separate optimizer (like Adam) that then actually uses those gradients to update the weights.
✓ Quick Self-Check
1. What happens during the forward pass?
Input data flows through the network's layers, producing a final prediction.
Tap to reveal / hide
2. What does the loss function measure?
How wrong the prediction was compared to the true answer — the "regret."
Tap to reveal / hide
3. What mathematical rule does backpropagation use to assign blame to each weight?
The chain rule of calculus.
Tap to reveal / hide
4. Does backpropagation itself update the weights?
No — it only calculates the gradients (blame); the optimizer (like Adam) uses those gradients to actually update the weights.
Tap to reveal / hide
5. What is the standard default optimizer, and what is its typical starting learning rate?
Adam, typically starting at a learning rate of 0.001.
Tap to reveal / hide