Step by Step
1
Forward pass — compute predictions and loss
Input data is fed through every layer of the network, producing a prediction, which is then compared against the true answer using a loss function to calculate how wrong the prediction was.
Example: an image is fed through all layers, producing a prediction of "82% confident this is a dog," and the loss function compares this to the true label.
2
Backward pass — compute blame via backpropagation
The error (loss) flows backward through the network, using the chain rule of calculus to calculate exactly how much each individual weight contributed to that error.
Example: backpropagation determines that a specific weight deep in the network contributed heavily to the wrong prediction, while another weight barely mattered.
3
Weight update — adjust weights to reduce error
Each weight is nudged slightly in the direction that would reduce the error, scaled by the learning rate, using an optimizer like Adam or AdamW.
Example: a weight identified as contributing heavily to the error gets adjusted more substantially than a weight that barely contributed.
Applied Walkthrough
1
A single training step begins with a batch of images being fed through the network.
2
The forward pass produces predictions for each image, and the loss function calculates how wrong those predictions were compared to the true labels.
3
The backward pass then flows that error backward through every layer, calculating exactly how much each weight throughout the network contributed to the mistakes.
4
Finally, every weight gets updated slightly in the direction that reduces the error, and this entire three-step cycle repeats with the next batch of data — over and over, often millions of times across the full training process.
Exam Application
Exams frequently ask you to correctly order these three phases (forward pass, backward pass, weight update) or identify which phase a described action belongs to. This three-step cycle is the single most fundamental concept in how any neural network actually learns, making it one of the most universally tested topics across neural network courses.
⚠ Common Trap
The most common trap is conflating the backward pass (which only calculates the gradients — the "blame" for each weight) with the weight update itself (which actually changes the weights using those calculated gradients). These are two distinct, sequential steps, not one combined step.
✓ Quick Self-Check
1. What happens during the forward pass?
Input data flows through the network, producing a prediction, and the loss function measures how wrong that prediction was.
Tap to reveal / hide
2. What happens during the backward pass?
The error flows backward through the network via the chain rule, calculating how much each weight contributed to the mistake.
Tap to reveal / hide
3. What happens during the weight update step?
Each weight is adjusted slightly in the direction that reduces the error, scaled by the learning rate.
Tap to reveal / hide
4. Does the backward pass itself change the weights?
No — it only calculates the gradients (blame); the separate weight update step is what actually changes the weights.
Tap to reveal / hide
5. Roughly how many times does this three-step training cycle repeat during a full training process?
Often millions of times, across many batches and epochs of data.
Tap to reveal / hide