XGBoost: What It Is, How It Works, and How to Use It

XGBoost (eXtreme Gradient Boosting) is an open-source machine learning library that implements gradient-boosted decision trees. It is used for classification, regression and ranking on structured (tabular) data. Trees are added one at a time, and each new tree corrects the errors of the ensemble built so far.

Originally created by Tianqi Chen as a research project around 2014, XGBoost quickly became a standard tool for data scientists. It was detailed in the paper “XGBoost: A Scalable Tree Boosting System” by Tianqi Chen and Carlos Guestrin, presented at KDD 2016. Released under the Apache-2.0 license and developed by the open-source DMLC community, it offers interfaces for Python, R, Java, Scala, Julia, C++, and a command-line tool. It provides a scikit-learn compatible API (XGBClassifier, XGBRegressor) and integrates with Apache Spark and Dask, making it versatile across different environments.

How gradient boosting works

To understand XGBoost, you first need to understand gradient boosting. This is an ensemble method where decision trees are built one after another. Each new tree is trained to correct the errors of the ensemble built so far. Specifically, it fits the gradient of the loss function with respect to the current predictions.

The final prediction is the sum of all trees’ outputs, each scaled by the learning rate. This sequential approach allows the model to gradually reduce bias and improve accuracy. This contrasts with bagging methods, such as random forest, which build trees independently and average them. While bagging reduces variance, boosting focuses on reducing bias by correcting previous mistakes.

What XGBoost adds

XGBoost improves on plain gradient boosting in several key ways. It introduces a regularized objective to control overfitting. This includes L1 (alpha) and L2 (lambda) penalties on leaf weights, plus gamma, which sets the minimum loss reduction required to make a split. These features help the model generalize better to unseen data.

The algorithm also uses a second-order approximation of the loss function, using both the gradient and the Hessian. This allows for more precise and faster convergence. XGBoost also features sparsity-aware split finding. For missing values, it learns a default direction at each split, so missing data does not need to be imputed first.

Additional performance enhancements include a weighted quantile sketch for fast approximate split finding. It supports parallel split finding within each tree, cache-aware access, and out-of-core computation for data larger than memory. Recent versions also offer a histogram-based tree method (“hist”) and GPU training (device=“cuda”).

Key hyperparameters

Tuning hyperparameters is crucial for XGBoost performance. Here are the key settings and their defaults:

n_estimators: Number of boosting rounds (trees).
learning_rate (eta): Shrinks each tree’s contribution. Default is 0.3. Lower values require more trees.
max_depth: Maximum depth of each tree. Default is 6.
subsample: Fraction of rows sampled for each tree.
colsample_bytree: Fraction of columns sampled for each tree.
min_child_weight: Minimum sum of instance weight (Hessian) needed in a child.
reg_lambda: L2 regularization on leaf weights. Default is 1.
reg_alpha: L1 regularization on leaf weights. Default is 0.
gamma: Minimum loss reduction to split. Default is 0.

You can also use early stopping to stop adding trees when the validation metric stops improving for a set number of rounds, which limits overfitting without guessing the right number of trees in advance.

XGBoost in Python

Implementing XGBoost in Python is straightforward using the xgboost library. The following code demonstrates how to set up a classifier with common hyperparameters and early stopping.

from xgboost import XGBClassifier

model = XGBClassifier(
    n_estimators=500,
    learning_rate=0.05,
    max_depth=6,
    subsample=0.8,
    colsample_bytree=0.8,
    early_stopping_rounds=50,
)
model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], verbose=False)
preds = model.predict(X_test)

In recent XGBoost versions, early_stopping_rounds is passed directly to the model constructor, as shown above. This approach integrates well with standard machine learning workflows.

XGBoost vs random forest

The main difference between XGBoost and random forest lies in how trees are built. Random forest uses bagging: trees are built independently on bootstrap samples and then averaged. It usually works reasonably well with default settings and is less sensitive to hyperparameter tuning.

XGBoost uses boosting: trees are built sequentially. It is often more accurate on tabular data after tuning, but it is more sensitive to hyperparameter tuning. If you use too many trees or a high learning rate, XGBoost can overfit noisy data. For structured data, XGBoost is often the preferred choice when you have the time to tune it.

When to use it and when not

XGBoost is a strong choice and a common baseline for tabular data. It is widely used for fraud detection, credit scoring, churn prediction, click-through-rate prediction, demand forecasting with engineered features, and learning to rank. It has been widely used by winning teams in machine learning competitions such as Kaggle.

It is less suited to raw images, audio, or free text, where deep neural networks usually perform better. If you need alternatives in the same family, consider LightGBM (Microsoft) and CatBoost (Yandex), both released in 2017. These libraries offer different trade-offs in speed and accuracy.

FAQ

What is XGBoost used for?

XGBoost is used for classification, regression, and ranking tasks on structured or tabular data. It is a popular choice for competitive machine learning and production models.

Is XGBoost better than random forest?

XGBoost often achieves higher accuracy on tabular data but requires more tuning. Random forest is easier to use with defaults and less prone to overfitting out of the box.

How does XGBoost handle missing values?

XGBoost uses sparsity-aware split finding. It learns a default direction for missing values at each split, so you do not need to impute missing data before training.