Random Forest vs XGBoost: Differences, Use Cases & When to Use Each

Random Forest builds many decision trees independently on random samples of your data and averages their votes. XGBoost builds trees one after another, with each new tree correcting the errors left by the earlier ones. Choose Random Forest for a stable baseline with little tuning. Choose XGBoost when accuracy on tabular data matters most and you can tune it.

Why does this comparison matter?

You trained a Random Forest on a churn or loan default dataset and the results look reasonable. Someone on the team says XGBoost is what most winning tabular solutions use, and now you have to decide whether switching is worth the extra tuning and maintenance.

In our previous article, Decision Tree vs Random Forest, we saw that a single tree overfits easily and that averaging many trees fixes much of that. This article covers the other way of combining trees, and when it pays off.

A few terms first. An ensemble is a model made by combining many smaller models. Tabular data is data in rows and columns, like a customer table. ROC AUC is a score from 0.5 (random guessing) to 1.0 (perfect ranking) that measures how well a model separates two classes.

What is the main difference between Random Forest and XGBoost?

The main difference is how the trees are combined. Random Forest uses bagging: each tree is trained on its own random sample of rows and features, and the trees never see each other. XGBoost uses boosting: trees are added one at a time, and each new tree is trained to fix the mistakes the current ensemble still makes.

Two terms from the table. Bias is error from a model being too simple to capture the pattern. Variance is error from a model being too sensitive to the particular training sample. Deep trees have low bias and high variance, so averaging them (Random Forest) cuts variance. Shallow trees have high bias, so adding them one by one (XGBoost) cuts bias.

How does Random Forest work?

Random Forest trains each tree on a bootstrap sample, which is a random sample of your rows drawn with replacement. At every split, a tree may only consider a random subset of the features. For classification, the forest takes a majority vote across trees. For regression, it averages them. Leo Breiman introduced the method in 2001.

The two sources of randomness make the trees different from each other, and averaging different trees cancels out much of their individual noise. Rows left out of each bootstrap sample give a free validation estimate, called the out-of-bag score (oob_score=True in scikit-learn).

Adding more trees to a Random Forest does not by itself cause overfitting. The error levels off, so more trees cost only training time.

How does XGBoost work?

XGBoost starts with a simple base prediction, measures where it is wrong, and trains a small tree to correct those errors. It adds that tree to the model, scaled down by a learning rate (a number below 1 that controls how much each tree is allowed to change the answer), and repeats.

The "gradient" in gradient boosting means each new tree is fitted to the direction in which the loss function (the measure of error) falls fastest. XGBoost, created by Tianqi Chen and Carlos Guestrin and published at KDD 2016, uses both the first and second derivatives of the loss to choose splits. It also adds regularisation, which means penalties on the number of leaves and the size of leaf values, to keep the trees from fitting noise.

Three practical features matter in day-to-day work:

• Early stopping. You give XGBoost a validation set and it stops adding trees when the score stops improving.
• Native handling of missing values. At each split it learns which direction rows with missing values should go.
• Imbalance support. The scale_pos_weight setting adjusts for rare positive classes.

Which hyperparameters matter for each?

Both models are controlled by a handful of settings. In XGBoost, the interaction between learning rate and number of trees is the one to understand first. A lower learning rate needs more trees.

For a deeper look at fixing overfitting in tree models, see Overfitting in Decision Trees and Random Forests.

Does XGBoost always beat Random Forest? A tested example

No. In our test, XGBoost scored clearly higher on a dataset with complex interactions between features and tied with Random Forest on a noisy one. We tested both on two synthetic datasets built with scikit-learn's make_classification, with 10,000 rows each and an 80/20 train-test split. Both models used 300 trees and default settings, with no tuning. We ran each dataset with five random seeds. The software was scikit-learn 1.8.0 and XGBoost 3.4.1.

On dataset A, the two models were level, and each won on some seeds. On dataset B, XGBoost scored higher on all five seeds. This is synthetic data, so treat it as an illustration of when the gap appears, not as a claim about your data.

You can reproduce dataset B with this code. With seed 42 it prints about 0.942 for Random Forest and about 0.957 for XGBoost.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from xgboost import XGBClassifier

X, y = make_classification(n_samples=10000, n_features=30, n_informative=15,
                           n_redundant=0, n_clusters_per_class=4,
                           flip_y=0.01, class_sep=1.0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

models = {
    "Random Forest": RandomForestClassifier(n_estimators=300, random_state=42, n_jobs=-1),
    "XGBoost": XGBClassifier(n_estimators=300, random_state=42, n_jobs=-1),
}
for name, model in models.items():
    model.fit(X_train, y_train)
    auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
    print(name, round(auc, 4))

When should you use Random Forest?

Use Random Forest when you want a reliable result quickly and do not want to spend time tuning. It suits these situations:

• You need a baseline. Default settings usually give a sensible first result, which tells you whether a more complex model is worth trying.
• Your labels are noisy. Averaging independent trees is forgiving of mislabelled rows.
• You want a validation estimate without a separate split. The out-of-bag score gives one.
• Your team has limited time for tuning. Few settings change the outcome much.

When should you use XGBoost?

Use XGBoost when the model's accuracy has real business value and you can afford to tune it properly. It suits these situations:

• The data has complex interactions between features. Sequential correction finds patterns that averaging can miss, as dataset B showed.
• You have missing values. XGBoost handles them without imputation.
• Classes are imbalanced. Credit default, fraud and churn problems usually are.
• You can run a proper validation setup. Early stopping depends on it.
• You need a custom loss function. XGBoost lets you supply your own objective.

When should you not use either?

You may not need either model. Check these cases before reaching for an ensemble:

• You must explain every prediction. In a regulated setting such as credit or insurance, a logistic regression or a single shallow decision tree is easier to defend than 300 trees. SHAP values help explain tree ensembles, but they are an approximation of the model's behaviour, not a rule you can hand to an auditor.
• Random Forest is already good enough. If the gain from XGBoost is small, as on dataset A, the extra tuning and monitoring may not be worth it.
• You need to extrapolate. Tree models cannot predict beyond the range of values seen in training. A sales series with a rising trend needs a different approach, or a trend removed first.
• Your data is images, audio or free text. Deep learning or pretrained models usually do better there.
• The dataset is very small. With a few hundred rows, a simpler model often generalises as well, and any ensemble comparison will be noisy.

What should you do next?

1. Run both on your own data with the same split and metric. Start with default settings, as in the code above. Use cross-validation, not one split, so you can see how much the scores vary.
2. Tune XGBoost with early stopping. Set learning_rate between 0.05 and 0.1, a high n_estimators, and try max_depth from 3 to 8. Tune Random Forest with min_samples_leaf and max_features.
3. Check what the model relies on before you decide. Look at feature importance and read Feature Importance in Random Forest: How It Works and Its Limitations, since impurity-based importance can mislead.

What comes next in this series

Random Forest and XGBoost are the two best-known examples of bagging and boosting. The next article, Bagging vs Boosting in Machine Learning, explains the idea behind both so you can reason about any ensemble. After that, XGBoost vs LightGBM vs CatBoost compares the three main boosting libraries.

Frequently asked questions

Is XGBoost always more accurate than Random Forest?

No. On tabular data XGBoost often scores higher once tuned, but on noisy data the two can tie, as in our dataset A. The only way to know is to test both on your data with the same split and metric.

Which overfits more, Random Forest or XGBoost?

XGBoost overfits more easily because it keeps fitting the remaining errors for as long as you add trees. Early stopping, a low learning rate and shallow trees control this. Random Forest is more forgiving, because adding trees does not by itself cause overfitting, although fully grown trees can still fit noise.

Do I need to scale or normalise features for these models?

No. Both are tree-based and split on thresholds, so the scale of a feature does not change the result. You still need to encode text categories as numbers for Random Forest in scikit-learn.

Which one handles imbalanced classes better?

Both can handle them. Random Forest has a class_weight setting and XGBoost has scale_pos_weight. For either, judge the model with ROC AUC or precision and recall rather than accuracy, which is misleading when one class is rare.

Can I use Random Forest and XGBoost together?

Yes. Averaging the predictions of both, or stacking them under another model, can give a small gain when the two make different errors. Test it on a validation set, as the gain is not guaranteed.

References

• Breiman, L. (2001). Random Forests. Machine Learning, 45, 5 to 32. https://doi.org/10.1023/A:1010933404324
• Chen, T. and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of KDD 2016. https://arxiv.org/abs/1603.02754
• XGBoost documentation, Introduction to Boosted Trees and Parameters. https://xgboost.readthedocs.io/en/stable/
• scikit-learn documentation, Ensembles: Gradient boosting, Random Forests, Bagging. https://scikit-learn.org/stable/modules/ensemble.html

Ivy Professional School has trained learners in data science and AI since 2008, with 37,500+ alumni and 400+ trainers. Explore Ivy Professional School's Data Science with Machine Learning & AI course.