Step by Step
B
Bagging — parallel, independent models reduce variance
Trains many models independently in parallel, each on a random subset of the data, then averages their predictions. This reduces variance. Random Forest is bagging applied specifically to decision trees.
Example: Random Forest training 300 decision trees independently on random subsets of the training data, then averaging their predictions.
O
(Boosting) — sequential models that correct previous errors
Trains models sequentially, where each new model specifically focuses on correcting the errors of the previous model. This reduces bias. XGBoost and LightGBM are popular gradient boosting implementations.
Example: XGBoost training a series of trees, where each new tree specifically targets the examples the previous trees got wrong.
S
Stacking — a meta-model combines base model outputs
Trains several different base models, then trains a separate meta-model whose job is to learn how to best combine those base models' outputs into a final prediction.
Example: training a logistic regression, a random forest, and a neural network as base models, then training a meta-model that learns how to weigh and combine their three separate predictions.
Applied Walkthrough
1
A data science team building a model for a Kaggle competition wants to maximize predictive accuracy, having already tried several individual models.
2
Ask: is a single, very carefully-tuned model usually the winning approach in these competitions? No — ensemble methods consistently outperform single models, which is exactly why they win most Kaggle competitions.
3
The team could try bagging (like Random Forest) for variance reduction, boosting (like XGBoost) for bias reduction, or even stacking multiple different model types together with a meta-model.
4
This reflects the No Free Lunch principle referenced in this topic: it's generally worth trying ensemble approaches before investing heavily in one single, increasingly complex model.
Exam Application
Exams test whether you can distinguish bagging (parallel, reduces variance, Random Forest) from boosting (sequential, reduces bias, XGBoost/LightGBM) and whether you understand stacking as a third, distinct ensemble approach that uses a meta-model to combine different base models.
⚠ Common Trap
The most common trap is treating bagging, boosting, and stacking as interchangeable synonyms for "ensemble." Each works through a fundamentally different mechanism — parallel and independent (bagging), sequential and error-correcting (boosting), or a learned combination via meta-model (stacking) — and mixing these up is a frequently tested mistake.
✓ Quick Self-Check
1. Does bagging train its models in parallel or sequentially?
In parallel, with each model trained independently on a random data subset.
Tap to reveal / hide
2. Does boosting train its models in parallel or sequentially?
Sequentially, with each new model correcting the previous model's errors.
Tap to reveal / hide
3. What is Random Forest an example of: bagging, boosting, or stacking?
Bagging — it's bagging applied specifically to decision trees.
Tap to reveal / hide
4. What does a meta-model do in stacking?
Learns how to best combine the outputs of several different base models into a final prediction.
Tap to reveal / hide
5. According to this lesson, do ensembles generally outperform single models?
Yes — ensembles consistently outperform single models, which is why they win most Kaggle competitions.
Tap to reveal / hide