Bagging vs Boosting in Machine Learning: Differences and When to Use Each

Why does the choice between bagging and boosting matter?

Most tabular machine learning problems in industry, such as credit risk, churn, demand forecasting and fraud, are solved with tree-based ensembles. Picking the wrong family costs you either accuracy or time. A boosted model on noisy data can chase errors that are not real. A bagged model on a problem that needs finer detail can plateau well below what is possible.

The two methods are often taught as a pair, but they solve different problems. Once you understand which problem each one targets, the choice becomes a diagnosis rather than a guess.

What is an ensemble, and why does it help?

An ensemble is a group of models whose predictions are combined into one final prediction. Combining models helps because a single model makes two kinds of error, and ensembles attack them differently.

• Bias is error from a model being too simple to capture the pattern. A shallow tree on a complex problem has high bias. It underfits.
• Variance is error from a model being too sensitive to the exact training sample. A deep, unpruned tree has high variance. It overfits, so a small change in the data gives a very different tree.

Bagging is mainly a variance-reduction technique. Boosting is mainly a bias-reduction technique. If you want a fuller treatment of the trade-off, read our explainer on the bias-variance trade-off.

How does bagging work?

Bagging, short for bootstrap aggregating, trains many models independently on different random samples of the training data and combines their outputs by averaging (regression) or majority vote (classification). Leo Breiman introduced it in 1996.

1. Draw a bootstrap sample from the training set. This means sampling rows at random with replacement, so some rows appear more than once and others not at all. A bootstrap sample contains roughly 63.2% of the distinct rows; the remaining 36.8% are left out.
2. Train one model, usually a deep decision tree, on that sample.
3. Repeat for 100 to 500 samples. Each model is trained independently, so the work can run in parallel across CPU cores.
4. Combine the predictions by averaging or voting.

The rows each model never saw are called out-of-bag rows. You can score each model on its own out-of-bag rows and get a free validation estimate without a separate hold-out set.

Random Forest is the best-known bagging method. It adds one more source of randomness: at every split, each tree considers only a random subset of the features. This keeps the trees from all looking alike, which makes averaging more effective.

How does boosting work?

Boosting trains models in sequence, where each new model is built to fix the mistakes of the ensemble so far, and the final prediction is a weighted sum of all models. The individual models are deliberately weak, meaning only slightly better than guessing. A common weak learner is a shallow tree with 1 to 6 levels.

There are two main flavours:

• AdaBoost (Freund and Schapire, 1997) increases the weight of rows the previous model got wrong, so the next model concentrates on them.
• Gradient boosting (Friedman, 2001) fits each new model to the residual errors of the current ensemble. Technically it takes a step along the negative gradient of the loss function.

Modern libraries build on gradient boosting. XGBoost (Chen and Guestrin, 2016) adds regularisation and a fast system design. LightGBM (Ke et al., 2017) uses histogram-based splits to train quickly on large data. CatBoost (Prokhorenkova et al., 2018) handles categorical features directly. Each also supports a learning rate, a small multiplier on every new model's contribution, which slows learning and reduces overfitting.

What are the differences between bagging and boosting?

Bagging builds independent models in parallel to reduce variance, while boosting builds dependent models in sequence to reduce bias. The table below sets out the practical differences.

Aspect	Bagging	Boosting
Training order	Parallel. Models are independent.	Sequential. Each model depends on the previous ones.
Main error reduced	Variance	Bias (and some variance)
Base learner	Strong, high-variance models, such as deep trees	Weak, high-bias models, such as shallow trees
Data used per model	Random bootstrap sample	Full data, with reweighted rows or residual targets
How models are combined	Equal-weight average or vote	Weighted sum
Overfitting risk	Low. More models rarely hurt.	Higher. Needs early stopping and a low learning rate.
Sensitivity to noisy labels	Low	High, because it keeps chasing hard rows
Tuning effort	Low. Defaults are often good.	Higher. Several interacting hyperparameters.
Training speed	Parallelises across models	Sequential across rounds, parallel inside each tree
Typical algorithms	Bagged trees, Random Forest, Extra Trees	AdaBoost, Gradient Boosting, XGBoost, LightGBM, CatBoost

Bagging

Training set

↓

Bootstrap sample 1
Tree 1

Bootstrap sample 2
Tree 2

Bootstrap sample 3
Tree 3

↓

Average or vote

Boosting

Tree 1

errors →

Tree 2

errors →

Tree 3

↓

Weighted sum

When should you use bagging?

Use bagging when your model overfits, meaning training accuracy is much higher than validation accuracy, and when you want a reliable result with little tuning. Specifically, bagging is a good fit when:

• The base model is unstable. Deep decision trees are the classic case.
• The data is noisy or has mislabelled rows. Averaging smooths out the noise instead of amplifying it.
• You need a strong baseline quickly. Random Forest with default settings is often within a few points of a tuned boosted model.
• You have many CPU cores and want training to scale across them.
• You want an out-of-bag error estimate without setting aside validation data.

Bagging does not help when the base model is stable and already low-variance, such as linear regression on a clean, well-specified problem. Averaging identical results gives you the same result.

When should you use boosting?

Use boosting when you need the highest accuracy on structured data and your model underfits, meaning even the training error is high. Boosting is a good fit when:

• You are competing on accuracy, such as a Kaggle problem or a production model where a fraction of a percentage point has real financial value.
• The data is clean enough that hard-to-predict rows are genuine signal and not label errors.
• Your classes are imbalanced. Boosting's focus on hard rows often helps with the minority class, though you should still evaluate with precision, recall and AUC, not accuracy alone.
• You can afford to tune the learning rate, number of trees, tree depth and regularisation, and to use early stopping on a validation set.

On tabular data, gradient boosting libraries are frequently the strongest option. This is a pattern seen across published benchmarks and competitions, not a law, so always confirm it on your own data.

Can you show the numbers behind each method?

Yes. Two small calculations show why bagging helps and what boosting does to the data. The numbers below are worked arithmetic for illustration, not results from a real dataset.

How do you compare both in code?

This scikit-learn snippet runs both families through the same cross-validation, so the comparison is fair. Replace the generated data with your own.

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

X, y = make_classification(n_samples=5000, n_features=20,
                           n_informative=8, random_state=42)

models = {
    "Bagging (Random Forest)": RandomForestClassifier(
        n_estimators=300, n_jobs=-1, random_state=42),
    "Boosting (Gradient Boosting)": GradientBoostingClassifier(
        n_estimators=300, learning_rate=0.05, max_depth=3, random_state=42),
}

for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
    print(f"{name}: mean AUC {scores.mean():.3f} (std {scores.std():.3f})")

Look at both the mean and the spread across folds. A slightly higher mean with a much larger spread is a warning sign. See our guide to cross-validation for how to read these numbers.

When should you not use bagging or boosting?

You may not need either. An ensemble is a tool for a specific problem, and several common situations do not call for one.

• A simple model already works. If logistic regression or a single shallow tree reaches the accuracy the business needs, stop there. A 300-tree ensemble adds cost, latency and maintenance for no practical gain.
• You must explain every prediction. A single tree or a linear model can be read line by line. Ensembles need tools such as SHAP to explain, and that is harder to defend in a regulated setting.
• The data is very small. With a few hundred rows, a boosted model tends to memorise the training data. A regularised simple model is usually safer.
• Labels are unreliable. Boosting keeps chasing hard rows, and wrong labels look exactly like hard rows. Clean the labels first, or prefer bagging.
• The data is images, audio or free text. Neural networks usually do better here. Tree ensembles are strongest on structured, tabular data.
• Prediction latency is tight. Hundreds of trees take longer to score than one model. Check your latency budget before you commit.

Also be careful with the idea that boosting is always better. It often wins on accuracy, but it also demands more tuning and more care. On a Tuesday afternoon with a deadline, a well-set Random Forest that you trust is often the better professional choice.

What should you do next?

1. Get a baseline. Train a single decision tree and a Random Forest on your data with default settings. Note training and validation scores.
2. Diagnose before you choose. If training is far above validation, you have a variance problem, so stay with bagging or add regularisation. If both are low, you have a bias problem, so try boosting.
3. Tune boosting with discipline. Fix a low learning rate (0.05 to 0.1), set a large number of trees, and use early stopping on a validation set. Then tune tree depth and regularisation. Our guide on hyperparameter tuning walks through the order.

If you want to practise this on real business datasets with a trainer reviewing your work, see how Random Forest and XGBoost compare in our Random Forest vs XGBoost guide. Ivy has been teaching data science and AI since 2008 and has trained 37,500+ alumni.

Frequently asked questions

Is Random Forest bagging or boosting?

Random Forest is a bagging method. It trains many decision trees in parallel on bootstrap samples and averages their predictions. It also picks a random subset of features at each split, which makes the trees less correlated with one another.

Is XGBoost bagging or boosting?

XGBoost is a boosting method. It builds trees one after another, and each new tree is fitted to correct the errors left by the trees before it. It adds regularisation and an efficient system design on top of standard gradient boosting.

Which is more accurate, bagging or boosting?

On structured, tabular data, well-tuned boosting usually reaches higher accuracy than bagging. The gap is not guaranteed, and boosting needs more tuning and is more sensitive to noisy labels. Always compare both with cross-validation on your own data.

Can bagging and boosting overfit?

Boosting overfits more easily because it keeps fitting the training errors, so it needs a low learning rate, limited tree depth and early stopping. Bagging rarely overfits as more models are added, but it cannot fix a model that is too simple for the problem.

Can I run bagging and boosting in parallel to save time?

Bagging trains each model independently, so it parallelises naturally across CPU cores. Boosting is sequential across rounds, though modern libraries parallelise the work inside each tree. For very large datasets, LightGBM and XGBoost are still fast in practice.

References

1. Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140. doi.org/10.1007/BF00058655
2. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. doi.org/10.1023/A:1010933404324
3. Freund, Y. and Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1), 119–139. doi.org/10.1006/jcss.1997.1504
4. Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine. Annals of Statistics, 29(5), 1189–1232. doi.org/10.1214/aos/1013203451
5. Chen, T. and Guestrin, C. (2016). XGBoost: a scalable tree boosting system. Proceedings of KDD 2016. arxiv.org/abs/1603.02754
6. Ke, G. et al. (2017). LightGBM: a highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems 30.
7. Prokhorenkova, L. et al. (2018). CatBoost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems 31.
8. scikit-learn documentation, Ensembles: Gradient boosting, forests of randomized trees, bagging and AdaBoost. scikit-learn.org/stable/modules/ensemble.html