How to Train a Gradient Boosting Regressor in Python

Build and evaluate a scikit-learn GradientBoostingRegressor on a synthetic dataset, printing test MSE and feature importances.

Medium Python 3.8+ Aug 9, 2026 ML engineering pipelines 13 views 0 copies

sklearn gradient-boosting regression machine-learning model-evaluation
Requires third-party packages — install first
pip install scikit-learn numpy

Python code

32 lines
Python 3.8+ 
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_squared_error

def train_gradient_boosting_mock():
    # Toy regression dataset
    np.random.seed(42)
    X = np.random.rand(100, 3) * 10
    y = 2 * X[:, 0] - 1.5 * X[:, 1] + 0.5 * X[:, 2] + np.random.normal(0, 1, 100)

    # Split into train/test
    split = int(0.8 * len(X))
    X_train, X_test = X[:split], X[split:]
    y_train, y_test = y[:split], y[split:]

    # Gradient boosting with modest hyperparameters
    model = GradientBoostingRegressor(
        n_estimators=100,
        learning_rate=0.1,
        max_depth=3,
        random_state=42
    )
    model.fit(X_train, y_train)

    # Predict and evaluate
    y_pred = model.predict(X_test)
    mse = mean_squared_error(y_test, y_pred)
    print(f"Test MSE: {mse:.4f}")
    print(f"Feature importances: {model.feature_importances_.round(4)}")

if __name__ == "__main__":
    train_gradient_boosting_mock()

Output

stdout 
Test MSE: 1.0662
Feature importances: [0.6954 0.2468 0.0578]

How it works

The GradientBoostingRegressor sequentially fits decision trees, where each new tree corrects the residual errors of the previous ensemble. With learning_rate=0.1, each tree contributes a small step, which reduces overfitting and typically improves generalization. The random_state=42 ensures reproducible results by fixing the stochastic elements (e.g., subsampling, feature selection). Feature importances reflect the average reduction in impurity contributed by each feature; here they show that feature 0 drives most of the target, consistent with the data-generating equation. The train/test split sets aside 20% of the samples to honestly estimate out-of-sample error — essential before trusting any metric.

Common mistakes

  • Not setting random_state, making results non-reproducible across runs.
  • Using default n_estimators=100 without tuning — try early stopping on a validation set.
  • Forgetting that split data must share the same ordering assumptions; always shuffle before splitting when data may be ordered.

Variations

  1. Use `HistGradientBoostingRegressor` from sklearn for faster training on large datasets.
  2. Use a validation set with `validation_fraction` and `n_iter_no_change` for automatic early stopping.

Real-world use cases

  • Predict house prices using structured features like area, bedrooms, and location coordinates.
  • Forecast demand for inventory planning from historic sales and promotional calendar features.
  • Estimate equipment maintenance costs from sensor readings and operational metadata in an IoT pipeline.

Sponsored

Run locally

This sample needs third-party packages, so it cannot run in the browser IDE. Copy the code above, install the packages shown at the top, then run it in your own Python environment.

More from ML engineering pipelines

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes