Step by Step
B
Bagging — parallel, independent trees, reduces variance
Bootstrap Aggregating trains many models IN PARALLEL, each on a random subset of the data, then averages their predictions together. This reduces variance. Random Forest is the gold-standard implementation.
Example: 300 trees, each trained independently and simultaneously on different random samples of the data, with their final predictions averaged together.
B
Boosting — sequential trees, each correcting the last, reduces bias
Boosting trains models SEQUENTIALLY, where each new model specifically focuses on the examples the previous model got wrong. This reduces bias. XGBoost, LightGBM, and AdaBoost are the most powerful implementations.
Example: tree 2 is trained specifically to fix the mistakes tree 1 made, then tree 3 is trained to fix whatever mistakes remain after trees 1 and 2, and so on.
⚖
Choosing between them — speed/reliability vs. maximum accuracy
Random Forest (bagging) is the go-to choice when you need speed and reliability with less tuning required. XGBoost (boosting) is the go-to choice when you need maximum accuracy on tabular data and are willing to invest more time in tuning.
Example: a quick baseline model for a hackathon might use Random Forest, while a Kaggle competition aiming for the absolute best possible tabular-data accuracy would typically reach for XGBoost.
Applied Walkthrough
1
A team needs a reliable baseline model quickly, without extensive hyperparameter tuning, for a tabular dataset.
2
Ask: does this favor bagging or boosting? Bagging (Random Forest) — it's known for reliability and requiring less tuning to get solid results.
3
Later, the same team wants to squeeze out maximum possible accuracy for a competition, and is willing to spend significant time tuning.
4
This shifts the choice toward boosting (XGBoost) — since sequential error-correction can often achieve higher accuracy than bagging, at the cost of more careful tuning and longer training time.
Exam Application
This bagging vs. boosting distinction is extremely heavily tested. Memorize the core contrast cold: bagging trains models in PARALLEL and independently (reduces variance, Random Forest); boosting trains models SEQUENTIALLY with each correcting the last (reduces bias, XGBoost/LightGBM/AdaBoost).
⚠ Common Trap
The most common trap is assuming bagging and boosting are just two names for the same general "ensemble" idea. They work through fundamentally different mechanisms — parallel and independent (bagging) vs. sequential and error-correcting (boosting) — and they target different problems (variance vs. bias respectively). Mixing these up is one of the single most common ML exam mistakes.
✓ Quick Self-Check
1. Does bagging train its models in parallel or sequentially?
In parallel — each model trains independently on a random subset of the data.
Tap to reveal / hide
2. Does boosting train its models in parallel or sequentially?
Sequentially — each new model specifically targets the previous model's mistakes.
Tap to reveal / hide
3. Which problem does bagging primarily reduce: bias or variance?
Variance.
Tap to reveal / hide
4. Which problem does boosting primarily reduce: bias or variance?
Bias.
Tap to reveal / hide
5. Name the gold-standard algorithm for bagging, and two popular algorithms for boosting.
Random Forest for bagging; XGBoost, LightGBM, or AdaBoost for boosting.
Tap to reveal / hide