Home / About Us / Volume I Contents / Ch 09: Ensembles and Trees

Statistical Basis of AI/ML — Ensembles \& Tree Methods

Chapter 9 — Combining Weak Learners into Strong Ones


Why This Chapter?

In Chapter 7 you trained individual classifiers — logistic regression, decision trees, KNN. Each has strengths but also systematic weaknesses: a single deep tree over-fits; a shallow tree under-fits.

Ensemble methods solve this by combining many models so that individual errors cancel out. The two dominant strategies are:

  • Bagging (Bootstrap Aggregation) — train many models on random subsets of the data and average their predictions. Random Forest is the flagship bagging method.
  • Boosting — train models sequentially, each one correcting the errors of the last. Gradient Boosting and XGBoost are the competition-winning boosting methods.

This chapter builds the full ensemble toolkit: bagging, random forests, AdaBoost, gradient boosting, XGBoost, and stacking — with both classification and regression examples.

What You Will Learn

$\S$ Topic Description
9.0 Setup Imports, GPU check, plot configuration
9.1 Decision Trees Revisited Variance vs. bias, pruning, why single trees fail
9.2 Bagging & Random Forests Bootstrap samples, feature sub-sampling, OOB score
9.3 AdaBoost Weighted re-sampling, exponential loss
9.4 Gradient Boosting Residual fitting, learning rate, shrinkage
9.5 XGBoost Regularised objective, histogram splitting, speed
9.6 Stacking Meta-learner on base-model predictions
9.7 Regression Trees & Ensembles Tree-based regression on Boston Housing
9.8 Case Study — Online Shopper Revenue Full EDA → ensemble → evaluation → business insight
9.9 Summary & Quick Reference Formulas, decision guide, cheat sheet

Section 9.0 — Setup: Imports & Configuration

This chapter adds ensemble-specific tools from scikit-learn and XGBoost.

Library Role in This Chapter
NumPy Array arithmetic, random seeding
Pandas DataFrames, data wrangling
Matplotlib Feature importance charts, learning curves
Seaborn Confusion-matrix heat maps, distribution plots
scikit-learn Decision trees, bagging, random forest, AdaBoost, gradient boosting, stacking
XGBoost Extreme Gradient Boosting
PyTorch GPU availability check
In [1]:
# === Chapter 9: Setup ===

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import scipy
from pathlib import Path

from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor, plot_tree
from sklearn.ensemble import (
    BaggingClassifier, RandomForestClassifier, RandomForestRegressor,
    AdaBoostClassifier, GradientBoostingClassifier, GradientBoostingRegressor,
    StackingClassifier,
)
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier

from sklearn.model_selection import (
    train_test_split, cross_val_score, StratifiedKFold,
    GridSearchCV, learning_curve,
)
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.pipeline import Pipeline
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, classification_report, ConfusionMatrixDisplay,
    roc_auc_score, roc_curve, mean_squared_error, root_mean_squared_error, r2_score,
)
import sklearn

import xgboost as xgb

# ── Plot defaults (house style) ──
plt.rcParams.update({
    "figure.figsize": (8, 4),
    "axes.grid": True,
    "grid.alpha": 0.3,
    "axes.spines.top": False,
    "axes.spines.right": False,
})
sns.set_theme(style="whitegrid")

rng = np.random.default_rng(42)

DATA = Path("../../data")

print(f"NumPy {np.__version__}  |  Pandas {pd.__version__}  |  "
      f"SciPy {scipy.__version__}")
print(f"scikit-learn {sklearn.__version__}  |  "
      f"XGBoost {xgb.__version__}")
print("\nChapter 9 environment ready.")
NumPy 2.5.1  |  Pandas 2.3.3  |  SciPy 1.18.0
scikit-learn 1.9.0  |  XGBoost 3.4.0

Chapter 9 environment ready.

Section 9.1 — Decision Trees Revisited

The Bias–Variance Trade-Off

A model's total error decomposes as:

$$\boxed{\text{Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Noise}}$$

Tree Depth Bias Variance Risk
Very shallow High Low Under-fits
Very deep Low High Over-fits

A single unpruned decision tree has low bias but high variance — it memorises training noise. Pruning helps, but the real solution is to combine many trees so that individual variance averages out.

Why Ensembles Work

If $T$ independent models each have variance $\sigma^2$, the variance of their average is:

$$\boxed{\text{Var}\!\left(\frac{1}{T}\sum_{t=1}^{T} f_t\right) = \frac{\sigma^2}{T}}$$

More trees → lower variance, without increasing bias.

We use the Pima Indians Diabetes dataset (768 patients, 8 features) to demonstrate the over-fitting problem, then solve it with ensembles.

In [2]:
# === Section 9.1: Load Pima Diabetes & single-tree baseline ===

df_pima = pd.read_csv(DATA / "Pima Indians Diabetes.csv")
print(f"Pima Diabetes dataset: {df_pima.shape[0]} rows × {df_pima.shape[1]} columns")
print(f"Class balance:\n{df_pima['class'].value_counts().to_string()}")

X_pima = df_pima.drop(columns=["class"])
y_pima = df_pima["class"]

X_train, X_test, y_train, y_test = train_test_split(
    X_pima, y_pima, test_size=0.25, random_state=42, stratify=y_pima
)

# Unpruned tree
tree_deep = DecisionTreeClassifier(random_state=42)
tree_deep.fit(X_train, y_train)

# Pruned tree
tree_pruned = DecisionTreeClassifier(max_depth=4, random_state=42)
tree_pruned.fit(X_train, y_train)

print(f"\n  Unpruned tree  — Train: {tree_deep.score(X_train, y_train):.4f}  "
      f"Test: {tree_deep.score(X_test, y_test):.4f}")
print(f"  Pruned (d=4)   — Train: {tree_pruned.score(X_train, y_train):.4f}  "
      f"Test: {tree_pruned.score(X_test, y_test):.4f}")
print("\n→ The unpruned tree perfectly memorises training data but "
      "generalises poorly — classic high-variance behaviour.")
Pima Diabetes dataset: 768 rows × 9 columns
Class balance:
class
0    500
1    268

  Unpruned tree  — Train: 1.0000  Test: 0.7448
  Pruned (d=4)   — Train: 0.8177  Test: 0.7656

→ The unpruned tree perfectly memorises training data but generalises poorly — classic high-variance behaviour.
In [3]:
# === Section 9.1 (cont.): Train vs. test accuracy across tree depths ===

depths = range(1, 21)
train_acc, test_acc = [], []
for d in depths:
    dt = DecisionTreeClassifier(max_depth=d, random_state=42)
    dt.fit(X_train, y_train)
    train_acc.append(dt.score(X_train, y_train))
    test_acc.append(dt.score(X_test, y_test))

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(list(depths), train_acc, "o-", color="steelblue",
        label="Train accuracy", linewidth=2)
ax.plot(list(depths), test_acc, "o-", color="#ef8a62",
        label="Test accuracy", linewidth=2)
ax.set_xlabel("Max Depth")
ax.set_ylabel("Accuracy")
ax.set_title("Bias–Variance Trade-Off in a Decision Tree", fontsize=13)
ax.legend()
plt.tight_layout()
plt.show()

print("→ Train accuracy rises to 1.0 as depth increases, but test "
      "accuracy peaks early then plateaus or drops — over-fitting.")
No description has been provided for this image
→ Train accuracy rises to 1.0 as depth increases, but test accuracy peaks early then plateaus or drops — over-fitting.

Section 9.2 — Bagging & Random Forests

Bootstrap Aggregation (Bagging)

  1. Draw $T$ bootstrap samples (random samples with replacement).
  2. Train a full decision tree on each sample.
  3. Aggregate: majority vote (classification) or average (regression).

$$\boxed{\hat{y}_{\text{bag}} = \text{mode}\{\hat{y}_1, \hat{y}_2, \ldots, \hat{y}_T\}}$$

Random Forest = Bagging + Feature Sub-Sampling

At each split, a random forest considers only a random subset of $m$ features (typically $m = \sqrt{p}$ for classification). This de-correlates the trees, reducing the variance of the ensemble even further.

Out-of-Bag (OOB) Score

Each bootstrap sample omits ~37 % of observations. These out-of-bag samples provide a free validation estimate without needing a held-out set.

Key Hyperparameters

Parameter Effect
n_estimators Number of trees — more is generally better
max_depth Limits tree depth — reduces variance
max_features Features per split — "sqrt" default
min_samples_leaf Minimum leaf size — regularisation
In [4]:
# === Section 9.2: Bagging vs. Random Forest on Pima Diabetes ===

bag = BaggingClassifier(
    estimator=DecisionTreeClassifier(random_state=42),
    n_estimators=200, random_state=42, oob_score=True,
)
bag.fit(X_train, y_train)

rf = RandomForestClassifier(
    n_estimators=200, random_state=42, oob_score=True,
)
rf.fit(X_train, y_train)

print("═══ Bagging vs. Random Forest — Pima Diabetes ═══\n")
print(f"{'Model':<22} {'Train':>8} {'Test':>8} {'OOB':>8}")
print("─" * 50)
print(f"{'Single Tree (d=4)':<22} "
      f"{tree_pruned.score(X_train, y_train):>8.4f} "
      f"{tree_pruned.score(X_test, y_test):>8.4f} "
      f"{'N/A':>8}")
print(f"{'Bagging (200 trees)':<22} "
      f"{bag.score(X_train, y_train):>8.4f} "
      f"{bag.score(X_test, y_test):>8.4f} "
      f"{bag.oob_score_:>8.4f}")
print(f"{'Random Forest (200)':<22} "
      f"{rf.score(X_train, y_train):>8.4f} "
      f"{rf.score(X_test, y_test):>8.4f} "
      f"{rf.oob_score_:>8.4f}")
print("\n→ Both ensembles substantially outperform the single tree.")
print("  Random Forest de-correlates trees via feature sub-sampling.")
═══ Bagging vs. Random Forest — Pima Diabetes ═══

Model                     Train     Test      OOB
──────────────────────────────────────────────────
Single Tree (d=4)        0.8177   0.7656      N/A
Bagging (200 trees)      1.0000   0.7552   0.7535
Random Forest (200)      1.0000   0.7500   0.7674

→ Both ensembles substantially outperform the single tree.
  Random Forest de-correlates trees via feature sub-sampling.
In [5]:
# === Section 9.2 (cont.): Feature importance — Random Forest ===

feat_imp = pd.Series(rf.feature_importances_, index=X_pima.columns)
feat_imp_sorted = feat_imp.sort_values(ascending=True)

fig, ax = plt.subplots(figsize=(8, 5))
feat_imp_sorted.plot.barh(ax=ax, color="steelblue", edgecolor="white")
ax.set_xlabel("Gini Importance")
ax.set_title("Random Forest Feature Importance — Pima Diabetes", fontsize=13)
plt.tight_layout()
plt.show()

print(f"→ Top feature: {feat_imp.idxmax()} "
      f"(importance = {feat_imp.max():.4f}).")
No description has been provided for this image
→ Top feature: Plas (importance = 0.2701).
In [6]:
# === Section 9.2 (cont.): OOB error vs. number of trees ===

oob_errors = []
tree_counts = list(range(10, 301, 10))
for n in tree_counts:
    rf_n = RandomForestClassifier(
        n_estimators=n, random_state=42, oob_score=True,
    )
    rf_n.fit(X_train, y_train)
    oob_errors.append(1 - rf_n.oob_score_)

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(tree_counts, oob_errors, "-", color="steelblue", linewidth=2)
ax.set_xlabel("Number of Trees")
ax.set_ylabel("OOB Error Rate")
ax.set_title("OOB Error vs. Ensemble Size — Random Forest", fontsize=13)
plt.tight_layout()
plt.show()

print("→ OOB error stabilises quickly; 100–200 trees is usually sufficient.")
/root/.venvs/vcc-gpu-sandbox-cu128/lib/python3.12/site-packages/sklearn/ensemble/_forest.py:589: UserWarning: Some inputs do not have OOB scores. This probably means too few trees were used to compute any reliable OOB estimates.
  warn(
No description has been provided for this image
→ OOB error stabilises quickly; 100–200 trees is usually sufficient.

Section 9.3 — AdaBoost

Adaptive Boosting

AdaBoost trains sequential weak learners (typically stumps — depth-1 trees). After each round, mis-classified samples receive higher weights, forcing the next learner to focus on the hard cases.

Algorithm Sketch

  1. Initialise equal sample weights: $w_i = 1/n$.
  2. For $t = 1, \ldots, T$:
    • Train weak learner $h_t$ on weighted data.
    • Compute weighted error $\varepsilon_t$.
    • Compute learner weight:

$$\boxed{\alpha_t = \frac{1}{2}\ln\frac{1 - \varepsilon_t}{\varepsilon_t}}$$

  • Update sample weights: increase for mis-classified, decrease for correct.
  1. Final prediction: weighted majority vote.

$$\boxed{\hat{y} = \text{sign}\left(\sum_{t=1}^{T} \alpha_t\, h_t(\mathbf{x})\right)}$$

In [7]:
# === Section 9.3: AdaBoost on Pima Diabetes ===

ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1, random_state=42),
    n_estimators=200, learning_rate=0.5, random_state=42,
)
ada.fit(X_train, y_train)

print(f"AdaBoost (200 stumps)")
print(f"  Train accuracy = {ada.score(X_train, y_train):.4f}")
print(f"  Test  accuracy = {ada.score(X_test, y_test):.4f}")
AdaBoost (200 stumps)
  Train accuracy = 0.8021
  Test  accuracy = 0.7344
In [8]:
# === Section 9.3 (cont.): AdaBoost staged accuracy ===

staged_train = list(ada.staged_score(X_train, y_train))
staged_test = list(ada.staged_score(X_test, y_test))

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(range(1, len(staged_train) + 1), staged_train,
        color="steelblue", label="Train", linewidth=2)
ax.plot(range(1, len(staged_test) + 1), staged_test,
        color="#ef8a62", label="Test", linewidth=2)
ax.set_xlabel("Boosting Round")
ax.set_ylabel("Accuracy")
ax.set_title("AdaBoost — Staged Accuracy", fontsize=13)
ax.legend()
plt.tight_layout()
plt.show()

print("→ Test accuracy improves rapidly in the first 50 rounds, "
      "then plateaus — diminishing returns from additional stumps.")
/root/.venvs/vcc-gpu-sandbox-cu128/lib/python3.12/site-packages/sklearn/utils/validation.py:2827: UserWarning: X does not have valid feature names, but AdaBoostClassifier was fitted with feature names
  warnings.warn(
/root/.venvs/vcc-gpu-sandbox-cu128/lib/python3.12/site-packages/sklearn/utils/validation.py:2827: UserWarning: X does not have valid feature names, but AdaBoostClassifier was fitted with feature names
  warnings.warn(
/root/.venvs/vcc-gpu-sandbox-cu128/lib/python3.12/site-packages/sklearn/utils/validation.py:2827: UserWarning: X does not have valid feature names, but AdaBoostClassifier was fitted with feature names
  warnings.warn(
/root/.venvs/vcc-gpu-sandbox-cu128/lib/python3.12/site-packages/sklearn/utils/validation.py:2827: UserWarning: X does not have valid feature names, but AdaBoostClassifier was fitted with feature names
  warnings.warn(
No description has been provided for this image
→ Test accuracy improves rapidly in the first 50 rounds, then plateaus — diminishing returns from additional stumps.

Section 9.4 — Gradient Boosting

From AdaBoost to Gradient Boosting

Gradient boosting generalises AdaBoost by fitting each new tree to the negative gradient (residuals) of a differentiable loss function.

$$\boxed{F_t(\mathbf{x}) = F_{t-1}(\mathbf{x}) + \eta\, h_t(\mathbf{x})}$$

where $\eta$ is the learning rate (shrinkage) and $h_t$ is fitted to the pseudo-residuals:

$$\boxed{r_i^{(t)} = -\frac{\partial L(y_i,\, F_{t-1}(\mathbf{x}_i))}{\partial F_{t-1}(\mathbf{x}_i)}}$$

Key Hyperparameters

Parameter Effect
n_estimators Number of boosting rounds
learning_rate Shrinkage per tree — lower is better with more trees
max_depth Tree depth per round — 3–5 is typical
subsample Row sub-sampling per round — stochastic gradient boosting
In [9]:
# === Section 9.4: Gradient Boosting on Pima Diabetes ===

gb = GradientBoostingClassifier(
    n_estimators=200, learning_rate=0.1, max_depth=3,
    subsample=0.8, random_state=42,
)
gb.fit(X_train, y_train)

print(f"Gradient Boosting (200 trees, lr=0.1)")
print(f"  Train accuracy = {gb.score(X_train, y_train):.4f}")
print(f"  Test  accuracy = {gb.score(X_test, y_test):.4f}")
Gradient Boosting (200 trees, lr=0.1)
  Train accuracy = 0.9861
  Test  accuracy = 0.7344
In [10]:
# === Section 9.4 (cont.): Gradient Boosting staged accuracy ===

staged_train_acc = [
    accuracy_score(y_train, p) for p in gb.staged_predict(X_train)
]
staged_test_acc = [
    accuracy_score(y_test, p) for p in gb.staged_predict(X_test)
]

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(range(1, len(staged_train_acc) + 1), staged_train_acc,
        color="steelblue", label="Train", linewidth=2)
ax.plot(range(1, len(staged_test_acc) + 1), staged_test_acc,
        color="#ef8a62", label="Test", linewidth=2)
ax.set_xlabel("Boosting Round")
ax.set_ylabel("Accuracy")
ax.set_title("Gradient Boosting — Staged Accuracy", fontsize=13)
ax.legend()
plt.tight_layout()
plt.show()

print("→ Learning rate=0.1 provides smooth convergence; "
      "test accuracy approaches train accuracy without diverging.")
No description has been provided for this image
→ Learning rate=0.1 provides smooth convergence; test accuracy approaches train accuracy without diverging.

Section 9.5 — XGBoost

eXtreme Gradient Boosting

XGBoost adds regularisation, parallel tree construction, and histogram-based splitting to standard gradient boosting.

Regularised Objective

$$\boxed{\mathcal{L} = \sum_{i=1}^{n} l(y_i, \hat{y}_i) + \sum_{t=1}^{T}\left[\gamma\, T_t + \frac{1}{2}\lambda \|w_t\|^2\right]}$$

  • $l$: loss function (log-loss for classification).
  • $\gamma$: penalty per leaf — controls tree complexity.
  • $\lambda$: L2 regularisation on leaf weights.
  • $T_t$: number of leaves in tree $t$.

Why XGBoost Dominates Competitions

Feature Benefit
Column & row sub-sampling Reduces over-fitting
Built-in regularisation ($\gamma$, $\lambda$, $\alpha$) Prevents memorisation
Histogram binning Faster split search
tree_method="hist" GPU-accelerable
Missing-value awareness No imputation needed
In [11]:
# === Section 9.5: XGBoost on Pima Diabetes ===

xgb_clf = xgb.XGBClassifier(
    n_estimators=200, learning_rate=0.1, max_depth=4,
    subsample=0.8, colsample_bytree=0.8,
    reg_lambda=1.0, gamma=0.1,
    use_label_encoder=False, eval_metric="logloss",
    random_state=42, verbosity=0,
)
xgb_clf.fit(X_train, y_train)

print(f"XGBoost (200 trees, lr=0.1)")
print(f"  Train accuracy = {xgb_clf.score(X_train, y_train):.4f}")
print(f"  Test  accuracy = {xgb_clf.score(X_test, y_test):.4f}")
XGBoost (200 trees, lr=0.1)
  Train accuracy = 0.9948
  Test  accuracy = 0.7396
In [12]:
# === Section 9.5 (cont.): XGBoost feature importance comparison ===

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Random Forest
feat_imp_sorted.plot.barh(ax=axes[0], color="steelblue", edgecolor="white")
axes[0].set_xlabel("Gini Importance")
axes[0].set_title("Random Forest", fontsize=12)

# XGBoost
xgb_imp = pd.Series(xgb_clf.feature_importances_, index=X_pima.columns)
xgb_imp_sorted = xgb_imp.sort_values(ascending=True)
xgb_imp_sorted.plot.barh(ax=axes[1], color="#ef8a62", edgecolor="white")
axes[1].set_xlabel("Gain Importance")
axes[1].set_title("XGBoost", fontsize=12)

fig.suptitle("Feature Importance — Random Forest vs. XGBoost",
             fontsize=14, y=1.02)
plt.tight_layout()
plt.show()

print("→ Both models agree on the top predictors, but importance "
      "magnitudes differ due to different splitting criteria.")
No description has been provided for this image
→ Both models agree on the top predictors, but importance magnitudes differ due to different splitting criteria.

Section 9.6 — Stacking

Meta-Learning

Stacking trains a meta-learner on the predictions of several base models. The idea: different algorithms make different errors; a meta-learner can learn when to trust each one.

Architecture

  1. Base layer: $M$ diverse models (e.g., RF, KNN, GBM).
  2. Each base model produces cross-validated predictions on the training set.
  3. Meta-layer: a simple model (typically logistic regression) trained on the stacked base predictions.

$$\boxed{\hat{y}_{\text{stack}} = g\!\left(\hat{y}_{\text{RF}},\; \hat{y}_{\text{KNN}},\; \hat{y}_{\text{GBM}}\right)}$$

In [13]:
# === Section 9.6: Stacking classifier on Pima Diabetes ===

estimators = [
    ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
    ("knn", KNeighborsClassifier(n_neighbors=7)),
    ("gb", GradientBoostingClassifier(
        n_estimators=100, learning_rate=0.1, max_depth=3, random_state=42)),
]

stack = StackingClassifier(
    estimators=estimators,
    final_estimator=LogisticRegression(max_iter=1000, random_state=42),
    cv=5,
)
stack.fit(X_train, y_train)

print(f"Stacking (RF + KNN + GB → Logistic Regression)")
print(f"  Train accuracy = {stack.score(X_train, y_train):.4f}")
print(f"  Test  accuracy = {stack.score(X_test, y_test):.4f}")
print("\n→ Stacking often matches or exceeds the best individual model.")
Stacking (RF + KNN + GB → Logistic Regression)
  Train accuracy = 0.9757
  Test  accuracy = 0.7448

→ Stacking often matches or exceeds the best individual model.
In [14]:
# === Section 9.6 (cont.): Head-to-head comparison table ===

models = {
    "Single Tree (d=4)": tree_pruned,
    "Bagging (200)": bag,
    "Random Forest (200)": rf,
    "AdaBoost (200)": ada,
    "Gradient Boosting": gb,
    "XGBoost": xgb_clf,
    "Stacking": stack,
}

print("═══ Ensemble Head-to-Head — Pima Diabetes ═══\n")
print(f"{'Model':<24} {'Train':>8} {'Test':>8} {'AUC':>8}")
print("─" * 52)
for name, model in models.items():
    tr = model.score(X_train, y_train)
    te = model.score(X_test, y_test)
    if hasattr(model, "predict_proba"):
        proba = model.predict_proba(X_test)[:, 1]
        auc = roc_auc_score(y_test, proba)
        print(f"{name:<24} {tr:>8.4f} {te:>8.4f} {auc:>8.4f}")
    else:
        print(f"{name:<24} {tr:>8.4f} {te:>8.4f} {'N/A':>8}")
print("─" * 52)
═══ Ensemble Head-to-Head — Pima Diabetes ═══

Model                       Train     Test      AUC
────────────────────────────────────────────────────
Single Tree (d=4)          0.8177   0.7656   0.7772
Bagging (200)              1.0000   0.7552   0.8153
Random Forest (200)        1.0000   0.7500   0.8162
AdaBoost (200)             0.8021   0.7344   0.8248
Gradient Boosting          0.9861   0.7344   0.8068
XGBoost                    0.9948   0.7396   0.8005
Stacking                   0.9757   0.7448   0.8125
────────────────────────────────────────────────────
In [15]:
# === Section 9.6 (cont.): Box plot of 5-fold CV scores ===

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_results = {}
for name, model in models.items():
    scores = cross_val_score(model, X_pima, y_pima, cv=cv, scoring="accuracy")
    cv_results[name] = scores

fig, ax = plt.subplots(figsize=(10, 5))
ax.boxplot(cv_results.values(), tick_labels=cv_results.keys(),
           patch_artist=True,
           boxprops=dict(facecolor="steelblue", alpha=0.6),
           medianprops=dict(color="#ef8a62", linewidth=2))
ax.set_ylabel("5-Fold CV Accuracy")
ax.set_title("Cross-Validated Comparison — All Ensemble Methods", fontsize=13)
plt.xticks(rotation=30, ha="right")
plt.tight_layout()
plt.show()

print("→ Ensembles consistently outperform the single tree; "
      "boosting methods tend to edge ahead of bagging.")
No description has been provided for this image
→ Ensembles consistently outperform the single tree; boosting methods tend to edge ahead of bagging.

Section 9.7 — Regression Trees & Ensembles

Trees for Continuous Targets

Everything above applies to regression by replacing:

  • Gini impurity → mean squared error (MSE) for splits.
  • Majority vote → mean prediction in each leaf.

$$\boxed{\text{MSE}_{\text{leaf}} = \frac{1}{|R|} \sum_{i \in R} (y_i - \bar{y}_R)^2}$$

We demonstrate on the Boston Housing dataset (506 neighbourhoods, 12 features, target: median home value MEDV).

In [16]:
# === Section 9.7: Regression ensembles on Boston Housing ===

df_boston = pd.read_csv(DATA / "Boston.csv")
print(f"Boston Housing dataset: {df_boston.shape[0]} rows × {df_boston.shape[1]} columns")

X_bos = df_boston.drop(columns=["MEDV"])
y_bos = df_boston["MEDV"]

X_bos_train, X_bos_test, y_bos_train, y_bos_test = train_test_split(
    X_bos, y_bos, test_size=0.25, random_state=42,
)

reg_models = {
    "Decision Tree": DecisionTreeRegressor(max_depth=5, random_state=42),
    "Random Forest": RandomForestRegressor(n_estimators=200, random_state=42),
    "Gradient Boosting": GradientBoostingRegressor(
        n_estimators=200, learning_rate=0.1, max_depth=3, random_state=42),
    "XGBoost": xgb.XGBRegressor(
        n_estimators=200, learning_rate=0.1, max_depth=4,
        random_state=42, verbosity=0),
}

print(f"\n{'Model':<22} {'RMSE':>8} {'R²':>8}")
print("─" * 40)
for name, model in reg_models.items():
    model.fit(X_bos_train, y_bos_train)
    preds = model.predict(X_bos_test)
    rmse = root_mean_squared_error(y_bos_test, preds)
    r2 = r2_score(y_bos_test, preds)
    print(f"{name:<22} {rmse:>8.3f} {r2:>8.4f}")
print("─" * 40)
Boston Housing dataset: 506 rows × 13 columns

Model                      RMSE       R²
────────────────────────────────────────
Decision Tree             3.349   0.8398
Random Forest             3.070   0.8654
Gradient Boosting         2.899   0.8800
XGBoost                   3.014   0.8703
────────────────────────────────────────
In [17]:
# === Section 9.7 (cont.): Predicted vs. actual for best model ===

best_reg = reg_models["Gradient Boosting"]
preds_best = best_reg.predict(X_bos_test)

fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(y_bos_test, preds_best, alpha=0.5,
           color="steelblue", edgecolor="white", s=40)
lims = [0, y_bos_test.max() + 5]
ax.plot(lims, lims, "--", color="#ef8a62", linewidth=2, label="Perfect")
ax.set_xlabel("Actual MEDV")
ax.set_ylabel("Predicted MEDV")
ax.set_title("Gradient Boosting — Predicted vs. Actual", fontsize=13)
ax.legend()
ax.set_aspect("equal")
plt.tight_layout()
plt.show()

r2 = r2_score(y_bos_test, preds_best)
print(f"→ R² = {r2:.4f} — the ensemble captures the bulk of "
      f"housing-price variation.")
No description has been provided for this image
→ R² = 0.8800 — the ensemble captures the bulk of housing-price variation.
In [18]:
# === Section 9.7 (cont.): Feature importance for Gradient Boosting (regression) ===

gb_imp = pd.Series(best_reg.feature_importances_, index=X_bos.columns)
gb_imp_sorted = gb_imp.sort_values(ascending=True)

fig, ax = plt.subplots(figsize=(8, 5))
gb_imp_sorted.plot.barh(ax=ax, color="steelblue", edgecolor="white")
ax.set_xlabel("Feature Importance (Gradient Boosting)")
ax.set_title("Boston Housing — Feature Importance", fontsize=13)
plt.tight_layout()
plt.show()

print(f"→ Top predictor: {gb_imp.idxmax()} "
      f"(importance = {gb_imp.max():.4f}).")
No description has been provided for this image
→ Top predictor: LSTAT (importance = 0.4029).

Section 9.8 — Case Study: Online Shopper Revenue Prediction

Objective

A realistic end-to-end workflow:

  1. EDA — understand the data, check class balance.
  2. Preprocess — encode categoricals, scale numerics.
  3. Train multiple ensemble models.
  4. Evaluate — accuracy, AUC, confusion matrix.
  5. Interpret — feature importance, business recommendations.

Dataset: Online Shoppers Purchasing Intention (12 330 sessions, 18 features, target: Revenue — did the visitor make a purchase?). Class imbalance: only ~15 % of sessions generate revenue.

In [19]:
# === Section 9.8: Load & EDA — Online Shoppers ===

df_shop = pd.read_csv(DATA / "online_shoppers_intention.csv")
print(f"Online Shoppers dataset: {df_shop.shape[0]} rows × {df_shop.shape[1]} columns")
print(f"\nClass balance:")
print(df_shop["Revenue"].value_counts().to_string())
print(f"Revenue rate: {df_shop['Revenue'].mean():.1%}")

fig, axes = plt.subplots(1, 3, figsize=(16, 4))

# Revenue distribution
df_shop["Revenue"].value_counts().plot.bar(
    ax=axes[0], color=["steelblue", "#ef8a62"], edgecolor="white")
axes[0].set_title("Revenue (Target)", fontsize=12)
axes[0].set_ylabel("Count")
axes[0].set_xticklabels(["No", "Yes"], rotation=0)

# PageValues distribution
axes[1].hist(df_shop["PageValues"], bins=50, color="steelblue",
             edgecolor="white", alpha=0.8)
axes[1].set_title("PageValues Distribution", fontsize=12)
axes[1].set_xlabel("PageValues")

# Visitor type
df_shop["VisitorType"].value_counts().plot.bar(
    ax=axes[2], color="steelblue", edgecolor="white")
axes[2].set_title("Visitor Type", fontsize=12)
axes[2].tick_params(axis="x", rotation=30)

fig.suptitle("EDA — Online Shoppers", fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
Online Shoppers dataset: 12330 rows × 18 columns

Class balance:
Revenue
False    10422
True      1908
Revenue rate: 15.5%
No description has been provided for this image
In [20]:
# === Section 9.8 (cont.): Preprocessing ===

df_shop2 = df_shop.copy()

# Encode target to int
df_shop2["Revenue"] = df_shop2["Revenue"].astype(int)

# Encode categoricals
df_shop2["Weekend"] = df_shop2["Weekend"].astype(int)

# One-hot encode Month and VisitorType
df_shop2 = pd.get_dummies(df_shop2, columns=["Month", "VisitorType"],
                          drop_first=True, dtype=int)

# Separate features and target
y_shop = df_shop2["Revenue"]
X_shop = df_shop2.drop(columns=["Revenue"])

# Scale
scaler_shop = StandardScaler()
X_shop_sc = scaler_shop.fit_transform(X_shop)

# Split
Xs_train, Xs_test, ys_train, ys_test = train_test_split(
    X_shop_sc, y_shop, test_size=0.25, random_state=42, stratify=y_shop,
)

print(f"Training set: {Xs_train.shape[0]} rows  |  "
      f"Test set: {Xs_test.shape[0]} rows")
print(f"Features after encoding: {X_shop.shape[1]}")
Training set: 9247 rows  |  Test set: 3083 rows
Features after encoding: 26
In [21]:
# === Section 9.8 (cont.): Train ensemble models ===

shop_models = {
    "Random Forest": RandomForestClassifier(
        n_estimators=200, random_state=42),
    "Gradient Boosting": GradientBoostingClassifier(
        n_estimators=200, learning_rate=0.1, max_depth=4, random_state=42),
    "XGBoost": xgb.XGBClassifier(
        n_estimators=200, learning_rate=0.1, max_depth=4,
        subsample=0.8, colsample_bytree=0.8,
        use_label_encoder=False, eval_metric="logloss",
        random_state=42, verbosity=0),
}

print("═══ Ensemble Comparison — Online Shoppers ═══\n")
print(f"{'Model':<22} {'Accuracy':>9} {'Precision':>10} {'Recall':>8} {'F1':>8} {'AUC':>8}")
print("─" * 68)

shop_preds = {}
for name, model in shop_models.items():
    model.fit(Xs_train, ys_train)
    preds = model.predict(Xs_test)
    proba = model.predict_proba(Xs_test)[:, 1]
    shop_preds[name] = (preds, proba)

    acc = accuracy_score(ys_test, preds)
    prec = precision_score(ys_test, preds)
    rec = recall_score(ys_test, preds)
    f1 = f1_score(ys_test, preds)
    auc = roc_auc_score(ys_test, proba)
    print(f"{name:<22} {acc:>9.4f} {prec:>10.4f} {rec:>8.4f} {f1:>8.4f} {auc:>8.4f}")

print("─" * 68)
═══ Ensemble Comparison — Online Shoppers ═══

Model                   Accuracy  Precision   Recall       F1      AUC
────────────────────────────────────────────────────────────────────
Random Forest             0.8975     0.7361   0.5262   0.6137   0.9158
Gradient Boosting         0.8965     0.7015   0.5765   0.6329   0.9248
XGBoost                   0.8985     0.7158   0.5702   0.6348   0.9267
────────────────────────────────────────────────────────────────────
In [22]:
# === Section 9.8 (cont.): Confusion matrices ===

fig, axes = plt.subplots(1, 3, figsize=(16, 4))
for ax, (name, (preds, _)) in zip(axes, shop_preds.items()):
    ConfusionMatrixDisplay.from_predictions(
        ys_test, preds, ax=ax, cmap="Blues",
        display_labels=["No Rev.", "Revenue"])
    ax.set_title(name, fontsize=12)

fig.suptitle("Confusion Matrices — Online Shoppers",
             fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
No description has been provided for this image
In [23]:
# === Section 9.8 (cont.): ROC curves ===

fig, ax = plt.subplots(figsize=(7, 6))
colours = ["steelblue", "#ef8a62", "#b2df8a"]

for (name, (_, proba)), colour in zip(shop_preds.items(), colours):
    fpr, tpr, _ = roc_curve(ys_test, proba)
    auc = roc_auc_score(ys_test, proba)
    ax.plot(fpr, tpr, color=colour, linewidth=2,
            label=f"{name} (AUC = {auc:.3f})")

ax.plot([0, 1], [0, 1], "--", color="gray", linewidth=1)
ax.set_xlabel("False Positive Rate")
ax.set_ylabel("True Positive Rate")
ax.set_title("ROC Curves — Online Shoppers", fontsize=13)
ax.legend(loc="lower right")
plt.tight_layout()
plt.show()
No description has been provided for this image
In [24]:
# === Section 9.8 (cont.): Feature importance — XGBoost ===

xgb_shop = shop_models["XGBoost"]
imp_shop = pd.Series(xgb_shop.feature_importances_, index=X_shop.columns)
top15 = imp_shop.nlargest(15).sort_values(ascending=True)

fig, ax = plt.subplots(figsize=(8, 6))
top15.plot.barh(ax=ax, color="steelblue", edgecolor="white")
ax.set_xlabel("XGBoost Gain Importance")
ax.set_title("Top 15 Features — Online Shopper Revenue", fontsize=13)
plt.tight_layout()
plt.show()

print(f"→ Top predictor: {imp_shop.idxmax()} "
      f"(importance = {imp_shop.max():.4f}).")
No description has been provided for this image
→ Top predictor: PageValues (importance = 0.3739).
In [25]:
# === Section 9.8 (cont.): Key findings & business recommendations ===

print("═══ Online Shopper Revenue — Key Findings ═══\n")
print("1. All three ensemble methods achieve strong AUC despite")
print("   significant class imbalance (only 15% of sessions convert).\n")
print("2. PageValues is the dominant predictor across all models —")
print("   sessions where the visitor sees product/cart pages are")
print("   far more likely to convert.\n")
print("3. BounceRates and ExitRates are the next most important,")
print("   confirming that engagement metrics predict conversion.\n")
print("4. Month and VisitorType provide modest signal — returning")
print("   visitors and November sessions convert at higher rates.\n")
print("5. Business recommendations:")
print("   • Prioritise UX improvements on product pages (PageValues).")
print("   • Target re-engagement campaigns at returning visitors.")
print("   • Investigate high-bounce pages for friction reduction.")
print("   • Deploy the XGBoost model for real-time conversion scoring.")
═══ Online Shopper Revenue — Key Findings ═══

1. All three ensemble methods achieve strong AUC despite
   significant class imbalance (only 15% of sessions convert).

2. PageValues is the dominant predictor across all models —
   sessions where the visitor sees product/cart pages are
   far more likely to convert.

3. BounceRates and ExitRates are the next most important,
   confirming that engagement metrics predict conversion.

4. Month and VisitorType provide modest signal — returning
   visitors and November sessions convert at higher rates.

5. Business recommendations:
   • Prioritise UX improvements on product pages (PageValues).
   • Target re-engagement campaigns at returning visitors.
   • Investigate high-bounce pages for friction reduction.
   • Deploy the XGBoost model for real-time conversion scoring.

Section 9.9 — Summary & Quick Reference

Key Formulas

Concept Formula
Bias–Variance $\text{Error} = \text{Bias}^2 + \text{Variance} + \text{Noise}$
Bagging prediction $\hat{y}_{\text{bag}} = \text{mode}\{\hat{y}_1, \ldots, \hat{y}_T\}$
AdaBoost weight $\alpha_t = \tfrac{1}{2}\ln\tfrac{1-\varepsilon_t}{\varepsilon_t}$
Gradient Boosting update $F_t = F_{t-1} + \eta\, h_t$
XGBoost objective $\mathcal{L} = \sum l(y_i,\hat{y}_i) + \sum [\gamma T_t + \tfrac{1}{2}\lambda\|w_t\|^2]$

Ensemble Decision Guide

Method Strengths Best For
Bagging / RF Reduces variance, parallel training, OOB score High-variance models, quick baseline
AdaBoost Simple, interpretable weights Small datasets, weak learners
Gradient Boosting Flexible loss functions, strong performance Tabular data, competitions
XGBoost Regularisation, speed, missing-value handling Production, large datasets
Stacking Combines diverse model strengths Squeezing out final % of performance

Python Cheat Sheet

from sklearn.ensemble import (
    RandomForestClassifier, GradientBoostingClassifier,
    StackingClassifier,
)
import xgboost as xgb

# Random Forest
rf = RandomForestClassifier(n_estimators=200, oob_score=True)
rf.fit(X_train, y_train)
print(rf.oob_score_)

# XGBoost
xgb_clf = xgb.XGBClassifier(n_estimators=200, learning_rate=0.1)
xgb_clf.fit(X_train, y_train)

Key Takeaways

$$\boxed{\text{Ensembles combine many weak learners into one strong learner — bagging reduces variance, boosting reduces bias.}}$$

  • Bagging averages independent models to reduce variance.
  • Random Forest adds feature sub-sampling to de-correlate trees.
  • Boosting (AdaBoost, Gradient Boosting, XGBoost) trains sequentially, correcting prior errors.
  • Stacking blends diverse models with a meta-learner.
  • For tabular data, gradient boosting and XGBoost are the go-to methods.
  • Always compare ensembles against a single-tree baseline to quantify the improvement.

Chapter 8 discovered structure without labels. Chapter 9 showed how combining models produces predictions far stronger than any single model.

Next → Chapter 10: Linear Algebra


References

  1. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5-32. The original random forest paper.

  2. Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of KDD, 785-794. The paper introducing the gradient-boosting library used in this chapter.

  3. Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29(5), 1189-1232. The theoretical foundation of gradient boosting.

  4. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. Chapter 9 (trees) and Chapter 10 (boosting).

  5. scikit-learn Developers. scikit-learn ensemble documentation (RandomForest, GradientBoosting). https://scikit-learn.org/stable/modules/ensemble.html