Step by Step
M
MSE (Mean Squared Error) — for regression, sensitive to outliers
Used for regression tasks predicting continuous values. Because errors are squared, large errors are penalized disproportionately, making MSE sensitive to outliers.
Example: predicting house prices, where a single wildly inaccurate prediction contributes a disproportionately large amount to the total MSE loss.
A
MAE (Mean Absolute Error) — for regression, robust to outliers
Also used for regression, but takes the absolute value of errors rather than squaring them, making it more robust to outliers than MSE.
Example: in a dataset with a few extreme outlier prices, MAE would be less distorted by those outliers than MSE.
B
Binary Cross-Entropy — paired with sigmoid, for binary classification
Used for binary classification tasks, always paired with a sigmoid output layer that produces a single probability between 0 and 1.
Example: a spam classifier outputting a single probability via sigmoid, trained using binary cross-entropy loss.
C
Categorical Cross-Entropy — paired with softmax, for multi-class classification
Used for multi-class classification tasks, always paired with a softmax output layer that produces a probability distribution across all classes summing to 1.
Example: a 10-digit classifier outputting a probability distribution across all 10 digits via softmax, trained using categorical cross-entropy loss.
Applied Walkthrough
1
A model is being built to classify images into one of 10 categories, and the developer must choose a loss function and output layer pairing.
2
Ask: is this binary or multi-class classification? Multi-class, since there are 10 categories.
3
That means the correct pairing is categorical cross-entropy loss with a softmax output layer, not binary cross-entropy with sigmoid.
4
Using the wrong pairing (say, sigmoid with categorical cross-entropy) would produce outputs that don't properly represent a valid probability distribution across the 10 classes, harming training.
Exam Application
Exams heavily test the correct pairing of loss function to output layer activation: binary cross-entropy always pairs with sigmoid, categorical cross-entropy always pairs with softmax. Also expect a distinction between loss (must be differentiable, optimized during training) and metric (what you actually report, like accuracy, F1, or AUC — doesn't need to be differentiable).
⚠ Common Trap
The most common trap is mismatching the loss function with the wrong output activation — for example, using categorical cross-entropy with a sigmoid output, or binary cross-entropy with a softmax output. These pairings must always match the task type (binary vs. multi-class) correctly.
✓ Quick Self-Check
1. Which loss function is more sensitive to outliers: MSE or MAE?
MSE — because it squares errors, large errors are penalized disproportionately.
Tap to reveal / hide
2. What output activation should always pair with binary cross-entropy?
Sigmoid.
Tap to reveal / hide
3. What output activation should always pair with categorical cross-entropy?
Softmax.
Tap to reveal / hide
4. What is the key requirement for a loss function that isn't required for a metric?
The loss function must be differentiable, since it's directly optimized during training; a metric (like accuracy or F1) doesn't need to be differentiable since it's just reported, not optimized.
Tap to reveal / hide
5. Would MSE or categorical cross-entropy be appropriate for predicting a continuous house price?
MSE (or MAE) — house price prediction is a regression task, not classification.
Tap to reveal / hide