Statistical Basis of AI/ML — Ensembles \& Tree Methods
Chapter 9 — Combining Weak Learners into Strong Ones
Why This Chapter?
In Chapter 7 you trained individual classifiers — logistic regression, decision trees, KNN. Each has strengths but also systematic weaknesses: a single deep tree over-fits; a shallow tree under-fits.
Ensemble methods solve this by combining many models so that individual errors cancel out. The two dominant strategies are:
- Bagging (Bootstrap Aggregation) — train many models on random subsets of the data and average their predictions. Random Forest is the flagship bagging method.
- Boosting — train models sequentially, each one correcting the errors of the last. Gradient Boosting and XGBoost are the competition-winning boosting methods.
This chapter builds the full ensemble toolkit: bagging, random forests, AdaBoost, gradient boosting, XGBoost, and stacking — with both classification and regression examples.
What You Will Learn
| $\S$ | Topic | Description |
|---|---|---|
| 9.0 | Setup | Imports, GPU check, plot configuration |
| 9.1 | Decision Trees Revisited | Variance vs. bias, pruning, why single trees fail |
| 9.2 | Bagging & Random Forests | Bootstrap samples, feature sub-sampling, OOB score |
| 9.3 | AdaBoost | Weighted re-sampling, exponential loss |
| 9.4 | Gradient Boosting | Residual fitting, learning rate, shrinkage |
| 9.5 | XGBoost | Regularised objective, histogram splitting, speed |
| 9.6 | Stacking | Meta-learner on base-model predictions |
| 9.7 | Regression Trees & Ensembles | Tree-based regression on Boston Housing |
| 9.8 | Case Study — Online Shopper Revenue | Full EDA → ensemble → evaluation → business insight |
| 9.9 | Summary & Quick Reference | Formulas, decision guide, cheat sheet |
Section 9.0 — Setup: Imports & Configuration
This chapter adds ensemble-specific tools from scikit-learn and XGBoost.
| Library | Role in This Chapter |
|---|---|
| NumPy | Array arithmetic, random seeding |
| Pandas | DataFrames, data wrangling |
| Matplotlib | Feature importance charts, learning curves |
| Seaborn | Confusion-matrix heat maps, distribution plots |
| scikit-learn | Decision trees, bagging, random forest, AdaBoost, gradient boosting, stacking |
| XGBoost | Extreme Gradient Boosting |
| PyTorch | GPU availability check |
# === Chapter 9: Setup ===
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import scipy
from pathlib import Path
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor, plot_tree
from sklearn.ensemble import (
BaggingClassifier, RandomForestClassifier, RandomForestRegressor,
AdaBoostClassifier, GradientBoostingClassifier, GradientBoostingRegressor,
StackingClassifier,
)
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import (
train_test_split, cross_val_score, StratifiedKFold,
GridSearchCV, learning_curve,
)
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.pipeline import Pipeline
from sklearn.metrics import (
accuracy_score, precision_score, recall_score, f1_score,
confusion_matrix, classification_report, ConfusionMatrixDisplay,
roc_auc_score, roc_curve, mean_squared_error, root_mean_squared_error, r2_score,
)
import sklearn
import xgboost as xgb
# ── Plot defaults (house style) ──
plt.rcParams.update({
"figure.figsize": (8, 4),
"axes.grid": True,
"grid.alpha": 0.3,
"axes.spines.top": False,
"axes.spines.right": False,
})
sns.set_theme(style="whitegrid")
rng = np.random.default_rng(42)
DATA = Path("../../data")
print(f"NumPy {np.__version__} | Pandas {pd.__version__} | "
f"SciPy {scipy.__version__}")
print(f"scikit-learn {sklearn.__version__} | "
f"XGBoost {xgb.__version__}")
print("\nChapter 9 environment ready.")
Section 9.1 — Decision Trees Revisited
The Bias–Variance Trade-Off¶
A model's total error decomposes as:
$$\boxed{\text{Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Noise}}$$
| Tree Depth | Bias | Variance | Risk |
|---|---|---|---|
| Very shallow | High | Low | Under-fits |
| Very deep | Low | High | Over-fits |
A single unpruned decision tree has low bias but high variance — it memorises training noise. Pruning helps, but the real solution is to combine many trees so that individual variance averages out.
Why Ensembles Work¶
If $T$ independent models each have variance $\sigma^2$, the variance of their average is:
$$\boxed{\text{Var}\!\left(\frac{1}{T}\sum_{t=1}^{T} f_t\right) = \frac{\sigma^2}{T}}$$
More trees → lower variance, without increasing bias.
We use the Pima Indians Diabetes dataset (768 patients, 8 features) to demonstrate the over-fitting problem, then solve it with ensembles.
# === Section 9.1: Load Pima Diabetes & single-tree baseline ===
df_pima = pd.read_csv(DATA / "Pima Indians Diabetes.csv")
print(f"Pima Diabetes dataset: {df_pima.shape[0]} rows × {df_pima.shape[1]} columns")
print(f"Class balance:\n{df_pima['class'].value_counts().to_string()}")
X_pima = df_pima.drop(columns=["class"])
y_pima = df_pima["class"]
X_train, X_test, y_train, y_test = train_test_split(
X_pima, y_pima, test_size=0.25, random_state=42, stratify=y_pima
)
# Unpruned tree
tree_deep = DecisionTreeClassifier(random_state=42)
tree_deep.fit(X_train, y_train)
# Pruned tree
tree_pruned = DecisionTreeClassifier(max_depth=4, random_state=42)
tree_pruned.fit(X_train, y_train)
print(f"\n Unpruned tree — Train: {tree_deep.score(X_train, y_train):.4f} "
f"Test: {tree_deep.score(X_test, y_test):.4f}")
print(f" Pruned (d=4) — Train: {tree_pruned.score(X_train, y_train):.4f} "
f"Test: {tree_pruned.score(X_test, y_test):.4f}")
print("\n→ The unpruned tree perfectly memorises training data but "
"generalises poorly — classic high-variance behaviour.")
# === Section 9.1 (cont.): Train vs. test accuracy across tree depths ===
depths = range(1, 21)
train_acc, test_acc = [], []
for d in depths:
dt = DecisionTreeClassifier(max_depth=d, random_state=42)
dt.fit(X_train, y_train)
train_acc.append(dt.score(X_train, y_train))
test_acc.append(dt.score(X_test, y_test))
fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(list(depths), train_acc, "o-", color="steelblue",
label="Train accuracy", linewidth=2)
ax.plot(list(depths), test_acc, "o-", color="#ef8a62",
label="Test accuracy", linewidth=2)
ax.set_xlabel("Max Depth")
ax.set_ylabel("Accuracy")
ax.set_title("Bias–Variance Trade-Off in a Decision Tree", fontsize=13)
ax.legend()
plt.tight_layout()
plt.show()
print("→ Train accuracy rises to 1.0 as depth increases, but test "
"accuracy peaks early then plateaus or drops — over-fitting.")
Section 9.2 — Bagging & Random Forests
Bootstrap Aggregation (Bagging)¶
- Draw $T$ bootstrap samples (random samples with replacement).
- Train a full decision tree on each sample.
- Aggregate: majority vote (classification) or average (regression).
$$\boxed{\hat{y}_{\text{bag}} = \text{mode}\{\hat{y}_1, \hat{y}_2, \ldots, \hat{y}_T\}}$$
Random Forest = Bagging + Feature Sub-Sampling¶
At each split, a random forest considers only a random subset of $m$ features (typically $m = \sqrt{p}$ for classification). This de-correlates the trees, reducing the variance of the ensemble even further.
Out-of-Bag (OOB) Score¶
Each bootstrap sample omits ~37 % of observations. These out-of-bag samples provide a free validation estimate without needing a held-out set.
Key Hyperparameters¶
| Parameter | Effect |
|---|---|
n_estimators |
Number of trees — more is generally better |
max_depth |
Limits tree depth — reduces variance |
max_features |
Features per split — "sqrt" default |
min_samples_leaf |
Minimum leaf size — regularisation |
# === Section 9.2: Bagging vs. Random Forest on Pima Diabetes ===
bag = BaggingClassifier(
estimator=DecisionTreeClassifier(random_state=42),
n_estimators=200, random_state=42, oob_score=True,
)
bag.fit(X_train, y_train)
rf = RandomForestClassifier(
n_estimators=200, random_state=42, oob_score=True,
)
rf.fit(X_train, y_train)
print("═══ Bagging vs. Random Forest — Pima Diabetes ═══\n")
print(f"{'Model':<22} {'Train':>8} {'Test':>8} {'OOB':>8}")
print("─" * 50)
print(f"{'Single Tree (d=4)':<22} "
f"{tree_pruned.score(X_train, y_train):>8.4f} "
f"{tree_pruned.score(X_test, y_test):>8.4f} "
f"{'N/A':>8}")
print(f"{'Bagging (200 trees)':<22} "
f"{bag.score(X_train, y_train):>8.4f} "
f"{bag.score(X_test, y_test):>8.4f} "
f"{bag.oob_score_:>8.4f}")
print(f"{'Random Forest (200)':<22} "
f"{rf.score(X_train, y_train):>8.4f} "
f"{rf.score(X_test, y_test):>8.4f} "
f"{rf.oob_score_:>8.4f}")
print("\n→ Both ensembles substantially outperform the single tree.")
print(" Random Forest de-correlates trees via feature sub-sampling.")
# === Section 9.2 (cont.): Feature importance — Random Forest ===
feat_imp = pd.Series(rf.feature_importances_, index=X_pima.columns)
feat_imp_sorted = feat_imp.sort_values(ascending=True)
fig, ax = plt.subplots(figsize=(8, 5))
feat_imp_sorted.plot.barh(ax=ax, color="steelblue", edgecolor="white")
ax.set_xlabel("Gini Importance")
ax.set_title("Random Forest Feature Importance — Pima Diabetes", fontsize=13)
plt.tight_layout()
plt.show()
print(f"→ Top feature: {feat_imp.idxmax()} "
f"(importance = {feat_imp.max():.4f}).")
# === Section 9.2 (cont.): OOB error vs. number of trees ===
oob_errors = []
tree_counts = list(range(10, 301, 10))
for n in tree_counts:
rf_n = RandomForestClassifier(
n_estimators=n, random_state=42, oob_score=True,
)
rf_n.fit(X_train, y_train)
oob_errors.append(1 - rf_n.oob_score_)
fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(tree_counts, oob_errors, "-", color="steelblue", linewidth=2)
ax.set_xlabel("Number of Trees")
ax.set_ylabel("OOB Error Rate")
ax.set_title("OOB Error vs. Ensemble Size — Random Forest", fontsize=13)
plt.tight_layout()
plt.show()
print("→ OOB error stabilises quickly; 100–200 trees is usually sufficient.")
Section 9.3 — AdaBoost
Adaptive Boosting¶
AdaBoost trains sequential weak learners (typically stumps — depth-1 trees). After each round, mis-classified samples receive higher weights, forcing the next learner to focus on the hard cases.
Algorithm Sketch¶
- Initialise equal sample weights: $w_i = 1/n$.
- For $t = 1, \ldots, T$:
- Train weak learner $h_t$ on weighted data.
- Compute weighted error $\varepsilon_t$.
- Compute learner weight:
$$\boxed{\alpha_t = \frac{1}{2}\ln\frac{1 - \varepsilon_t}{\varepsilon_t}}$$
- Update sample weights: increase for mis-classified, decrease for correct.
- Final prediction: weighted majority vote.
$$\boxed{\hat{y} = \text{sign}\left(\sum_{t=1}^{T} \alpha_t\, h_t(\mathbf{x})\right)}$$
# === Section 9.3: AdaBoost on Pima Diabetes ===
ada = AdaBoostClassifier(
estimator=DecisionTreeClassifier(max_depth=1, random_state=42),
n_estimators=200, learning_rate=0.5, random_state=42,
)
ada.fit(X_train, y_train)
print(f"AdaBoost (200 stumps)")
print(f" Train accuracy = {ada.score(X_train, y_train):.4f}")
print(f" Test accuracy = {ada.score(X_test, y_test):.4f}")
# === Section 9.3 (cont.): AdaBoost staged accuracy ===
staged_train = list(ada.staged_score(X_train, y_train))
staged_test = list(ada.staged_score(X_test, y_test))
fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(range(1, len(staged_train) + 1), staged_train,
color="steelblue", label="Train", linewidth=2)
ax.plot(range(1, len(staged_test) + 1), staged_test,
color="#ef8a62", label="Test", linewidth=2)
ax.set_xlabel("Boosting Round")
ax.set_ylabel("Accuracy")
ax.set_title("AdaBoost — Staged Accuracy", fontsize=13)
ax.legend()
plt.tight_layout()
plt.show()
print("→ Test accuracy improves rapidly in the first 50 rounds, "
"then plateaus — diminishing returns from additional stumps.")
Section 9.4 — Gradient Boosting
From AdaBoost to Gradient Boosting¶
Gradient boosting generalises AdaBoost by fitting each new tree to the negative gradient (residuals) of a differentiable loss function.
$$\boxed{F_t(\mathbf{x}) = F_{t-1}(\mathbf{x}) + \eta\, h_t(\mathbf{x})}$$
where $\eta$ is the learning rate (shrinkage) and $h_t$ is fitted to the pseudo-residuals:
$$\boxed{r_i^{(t)} = -\frac{\partial L(y_i,\, F_{t-1}(\mathbf{x}_i))}{\partial F_{t-1}(\mathbf{x}_i)}}$$
Key Hyperparameters¶
| Parameter | Effect |
|---|---|
n_estimators |
Number of boosting rounds |
learning_rate |
Shrinkage per tree — lower is better with more trees |
max_depth |
Tree depth per round — 3–5 is typical |
subsample |
Row sub-sampling per round — stochastic gradient boosting |
# === Section 9.4: Gradient Boosting on Pima Diabetes ===
gb = GradientBoostingClassifier(
n_estimators=200, learning_rate=0.1, max_depth=3,
subsample=0.8, random_state=42,
)
gb.fit(X_train, y_train)
print(f"Gradient Boosting (200 trees, lr=0.1)")
print(f" Train accuracy = {gb.score(X_train, y_train):.4f}")
print(f" Test accuracy = {gb.score(X_test, y_test):.4f}")
# === Section 9.4 (cont.): Gradient Boosting staged accuracy ===
staged_train_acc = [
accuracy_score(y_train, p) for p in gb.staged_predict(X_train)
]
staged_test_acc = [
accuracy_score(y_test, p) for p in gb.staged_predict(X_test)
]
fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(range(1, len(staged_train_acc) + 1), staged_train_acc,
color="steelblue", label="Train", linewidth=2)
ax.plot(range(1, len(staged_test_acc) + 1), staged_test_acc,
color="#ef8a62", label="Test", linewidth=2)
ax.set_xlabel("Boosting Round")
ax.set_ylabel("Accuracy")
ax.set_title("Gradient Boosting — Staged Accuracy", fontsize=13)
ax.legend()
plt.tight_layout()
plt.show()
print("→ Learning rate=0.1 provides smooth convergence; "
"test accuracy approaches train accuracy without diverging.")
Section 9.5 — XGBoost
eXtreme Gradient Boosting¶
XGBoost adds regularisation, parallel tree construction, and histogram-based splitting to standard gradient boosting.
Regularised Objective¶
$$\boxed{\mathcal{L} = \sum_{i=1}^{n} l(y_i, \hat{y}_i) + \sum_{t=1}^{T}\left[\gamma\, T_t + \frac{1}{2}\lambda \|w_t\|^2\right]}$$
- $l$: loss function (log-loss for classification).
- $\gamma$: penalty per leaf — controls tree complexity.
- $\lambda$: L2 regularisation on leaf weights.
- $T_t$: number of leaves in tree $t$.
Why XGBoost Dominates Competitions¶
| Feature | Benefit |
|---|---|
| Column & row sub-sampling | Reduces over-fitting |
| Built-in regularisation ($\gamma$, $\lambda$, $\alpha$) | Prevents memorisation |
| Histogram binning | Faster split search |
tree_method="hist" |
GPU-accelerable |
| Missing-value awareness | No imputation needed |
# === Section 9.5: XGBoost on Pima Diabetes ===
xgb_clf = xgb.XGBClassifier(
n_estimators=200, learning_rate=0.1, max_depth=4,
subsample=0.8, colsample_bytree=0.8,
reg_lambda=1.0, gamma=0.1,
use_label_encoder=False, eval_metric="logloss",
random_state=42, verbosity=0,
)
xgb_clf.fit(X_train, y_train)
print(f"XGBoost (200 trees, lr=0.1)")
print(f" Train accuracy = {xgb_clf.score(X_train, y_train):.4f}")
print(f" Test accuracy = {xgb_clf.score(X_test, y_test):.4f}")
# === Section 9.5 (cont.): XGBoost feature importance comparison ===
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Random Forest
feat_imp_sorted.plot.barh(ax=axes[0], color="steelblue", edgecolor="white")
axes[0].set_xlabel("Gini Importance")
axes[0].set_title("Random Forest", fontsize=12)
# XGBoost
xgb_imp = pd.Series(xgb_clf.feature_importances_, index=X_pima.columns)
xgb_imp_sorted = xgb_imp.sort_values(ascending=True)
xgb_imp_sorted.plot.barh(ax=axes[1], color="#ef8a62", edgecolor="white")
axes[1].set_xlabel("Gain Importance")
axes[1].set_title("XGBoost", fontsize=12)
fig.suptitle("Feature Importance — Random Forest vs. XGBoost",
fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
print("→ Both models agree on the top predictors, but importance "
"magnitudes differ due to different splitting criteria.")
Section 9.6 — Stacking
Meta-Learning¶
Stacking trains a meta-learner on the predictions of several base models. The idea: different algorithms make different errors; a meta-learner can learn when to trust each one.
Architecture¶
- Base layer: $M$ diverse models (e.g., RF, KNN, GBM).
- Each base model produces cross-validated predictions on the training set.
- Meta-layer: a simple model (typically logistic regression) trained on the stacked base predictions.
$$\boxed{\hat{y}_{\text{stack}} = g\!\left(\hat{y}_{\text{RF}},\; \hat{y}_{\text{KNN}},\; \hat{y}_{\text{GBM}}\right)}$$
# === Section 9.6: Stacking classifier on Pima Diabetes ===
estimators = [
("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
("knn", KNeighborsClassifier(n_neighbors=7)),
("gb", GradientBoostingClassifier(
n_estimators=100, learning_rate=0.1, max_depth=3, random_state=42)),
]
stack = StackingClassifier(
estimators=estimators,
final_estimator=LogisticRegression(max_iter=1000, random_state=42),
cv=5,
)
stack.fit(X_train, y_train)
print(f"Stacking (RF + KNN + GB → Logistic Regression)")
print(f" Train accuracy = {stack.score(X_train, y_train):.4f}")
print(f" Test accuracy = {stack.score(X_test, y_test):.4f}")
print("\n→ Stacking often matches or exceeds the best individual model.")
# === Section 9.6 (cont.): Head-to-head comparison table ===
models = {
"Single Tree (d=4)": tree_pruned,
"Bagging (200)": bag,
"Random Forest (200)": rf,
"AdaBoost (200)": ada,
"Gradient Boosting": gb,
"XGBoost": xgb_clf,
"Stacking": stack,
}
print("═══ Ensemble Head-to-Head — Pima Diabetes ═══\n")
print(f"{'Model':<24} {'Train':>8} {'Test':>8} {'AUC':>8}")
print("─" * 52)
for name, model in models.items():
tr = model.score(X_train, y_train)
te = model.score(X_test, y_test)
if hasattr(model, "predict_proba"):
proba = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, proba)
print(f"{name:<24} {tr:>8.4f} {te:>8.4f} {auc:>8.4f}")
else:
print(f"{name:<24} {tr:>8.4f} {te:>8.4f} {'N/A':>8}")
print("─" * 52)
# === Section 9.6 (cont.): Box plot of 5-fold CV scores ===
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_results = {}
for name, model in models.items():
scores = cross_val_score(model, X_pima, y_pima, cv=cv, scoring="accuracy")
cv_results[name] = scores
fig, ax = plt.subplots(figsize=(10, 5))
ax.boxplot(cv_results.values(), tick_labels=cv_results.keys(),
patch_artist=True,
boxprops=dict(facecolor="steelblue", alpha=0.6),
medianprops=dict(color="#ef8a62", linewidth=2))
ax.set_ylabel("5-Fold CV Accuracy")
ax.set_title("Cross-Validated Comparison — All Ensemble Methods", fontsize=13)
plt.xticks(rotation=30, ha="right")
plt.tight_layout()
plt.show()
print("→ Ensembles consistently outperform the single tree; "
"boosting methods tend to edge ahead of bagging.")
Section 9.7 — Regression Trees & Ensembles
Trees for Continuous Targets¶
Everything above applies to regression by replacing:
- Gini impurity → mean squared error (MSE) for splits.
- Majority vote → mean prediction in each leaf.
$$\boxed{\text{MSE}_{\text{leaf}} = \frac{1}{|R|} \sum_{i \in R} (y_i - \bar{y}_R)^2}$$
We demonstrate on the Boston Housing dataset (506 neighbourhoods, 12 features, target: median home value MEDV).
# === Section 9.7: Regression ensembles on Boston Housing ===
df_boston = pd.read_csv(DATA / "Boston.csv")
print(f"Boston Housing dataset: {df_boston.shape[0]} rows × {df_boston.shape[1]} columns")
X_bos = df_boston.drop(columns=["MEDV"])
y_bos = df_boston["MEDV"]
X_bos_train, X_bos_test, y_bos_train, y_bos_test = train_test_split(
X_bos, y_bos, test_size=0.25, random_state=42,
)
reg_models = {
"Decision Tree": DecisionTreeRegressor(max_depth=5, random_state=42),
"Random Forest": RandomForestRegressor(n_estimators=200, random_state=42),
"Gradient Boosting": GradientBoostingRegressor(
n_estimators=200, learning_rate=0.1, max_depth=3, random_state=42),
"XGBoost": xgb.XGBRegressor(
n_estimators=200, learning_rate=0.1, max_depth=4,
random_state=42, verbosity=0),
}
print(f"\n{'Model':<22} {'RMSE':>8} {'R²':>8}")
print("─" * 40)
for name, model in reg_models.items():
model.fit(X_bos_train, y_bos_train)
preds = model.predict(X_bos_test)
rmse = root_mean_squared_error(y_bos_test, preds)
r2 = r2_score(y_bos_test, preds)
print(f"{name:<22} {rmse:>8.3f} {r2:>8.4f}")
print("─" * 40)
# === Section 9.7 (cont.): Predicted vs. actual for best model ===
best_reg = reg_models["Gradient Boosting"]
preds_best = best_reg.predict(X_bos_test)
fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(y_bos_test, preds_best, alpha=0.5,
color="steelblue", edgecolor="white", s=40)
lims = [0, y_bos_test.max() + 5]
ax.plot(lims, lims, "--", color="#ef8a62", linewidth=2, label="Perfect")
ax.set_xlabel("Actual MEDV")
ax.set_ylabel("Predicted MEDV")
ax.set_title("Gradient Boosting — Predicted vs. Actual", fontsize=13)
ax.legend()
ax.set_aspect("equal")
plt.tight_layout()
plt.show()
r2 = r2_score(y_bos_test, preds_best)
print(f"→ R² = {r2:.4f} — the ensemble captures the bulk of "
f"housing-price variation.")
# === Section 9.7 (cont.): Feature importance for Gradient Boosting (regression) ===
gb_imp = pd.Series(best_reg.feature_importances_, index=X_bos.columns)
gb_imp_sorted = gb_imp.sort_values(ascending=True)
fig, ax = plt.subplots(figsize=(8, 5))
gb_imp_sorted.plot.barh(ax=ax, color="steelblue", edgecolor="white")
ax.set_xlabel("Feature Importance (Gradient Boosting)")
ax.set_title("Boston Housing — Feature Importance", fontsize=13)
plt.tight_layout()
plt.show()
print(f"→ Top predictor: {gb_imp.idxmax()} "
f"(importance = {gb_imp.max():.4f}).")
Section 9.8 — Case Study: Online Shopper Revenue Prediction
Objective¶
A realistic end-to-end workflow:
- EDA — understand the data, check class balance.
- Preprocess — encode categoricals, scale numerics.
- Train multiple ensemble models.
- Evaluate — accuracy, AUC, confusion matrix.
- Interpret — feature importance, business recommendations.
Dataset: Online Shoppers Purchasing Intention (12 330 sessions,
18 features, target: Revenue — did the visitor make a purchase?).
Class imbalance: only ~15 % of sessions generate revenue.
# === Section 9.8: Load & EDA — Online Shoppers ===
df_shop = pd.read_csv(DATA / "online_shoppers_intention.csv")
print(f"Online Shoppers dataset: {df_shop.shape[0]} rows × {df_shop.shape[1]} columns")
print(f"\nClass balance:")
print(df_shop["Revenue"].value_counts().to_string())
print(f"Revenue rate: {df_shop['Revenue'].mean():.1%}")
fig, axes = plt.subplots(1, 3, figsize=(16, 4))
# Revenue distribution
df_shop["Revenue"].value_counts().plot.bar(
ax=axes[0], color=["steelblue", "#ef8a62"], edgecolor="white")
axes[0].set_title("Revenue (Target)", fontsize=12)
axes[0].set_ylabel("Count")
axes[0].set_xticklabels(["No", "Yes"], rotation=0)
# PageValues distribution
axes[1].hist(df_shop["PageValues"], bins=50, color="steelblue",
edgecolor="white", alpha=0.8)
axes[1].set_title("PageValues Distribution", fontsize=12)
axes[1].set_xlabel("PageValues")
# Visitor type
df_shop["VisitorType"].value_counts().plot.bar(
ax=axes[2], color="steelblue", edgecolor="white")
axes[2].set_title("Visitor Type", fontsize=12)
axes[2].tick_params(axis="x", rotation=30)
fig.suptitle("EDA — Online Shoppers", fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
# === Section 9.8 (cont.): Preprocessing ===
df_shop2 = df_shop.copy()
# Encode target to int
df_shop2["Revenue"] = df_shop2["Revenue"].astype(int)
# Encode categoricals
df_shop2["Weekend"] = df_shop2["Weekend"].astype(int)
# One-hot encode Month and VisitorType
df_shop2 = pd.get_dummies(df_shop2, columns=["Month", "VisitorType"],
drop_first=True, dtype=int)
# Separate features and target
y_shop = df_shop2["Revenue"]
X_shop = df_shop2.drop(columns=["Revenue"])
# Scale
scaler_shop = StandardScaler()
X_shop_sc = scaler_shop.fit_transform(X_shop)
# Split
Xs_train, Xs_test, ys_train, ys_test = train_test_split(
X_shop_sc, y_shop, test_size=0.25, random_state=42, stratify=y_shop,
)
print(f"Training set: {Xs_train.shape[0]} rows | "
f"Test set: {Xs_test.shape[0]} rows")
print(f"Features after encoding: {X_shop.shape[1]}")
# === Section 9.8 (cont.): Train ensemble models ===
shop_models = {
"Random Forest": RandomForestClassifier(
n_estimators=200, random_state=42),
"Gradient Boosting": GradientBoostingClassifier(
n_estimators=200, learning_rate=0.1, max_depth=4, random_state=42),
"XGBoost": xgb.XGBClassifier(
n_estimators=200, learning_rate=0.1, max_depth=4,
subsample=0.8, colsample_bytree=0.8,
use_label_encoder=False, eval_metric="logloss",
random_state=42, verbosity=0),
}
print("═══ Ensemble Comparison — Online Shoppers ═══\n")
print(f"{'Model':<22} {'Accuracy':>9} {'Precision':>10} {'Recall':>8} {'F1':>8} {'AUC':>8}")
print("─" * 68)
shop_preds = {}
for name, model in shop_models.items():
model.fit(Xs_train, ys_train)
preds = model.predict(Xs_test)
proba = model.predict_proba(Xs_test)[:, 1]
shop_preds[name] = (preds, proba)
acc = accuracy_score(ys_test, preds)
prec = precision_score(ys_test, preds)
rec = recall_score(ys_test, preds)
f1 = f1_score(ys_test, preds)
auc = roc_auc_score(ys_test, proba)
print(f"{name:<22} {acc:>9.4f} {prec:>10.4f} {rec:>8.4f} {f1:>8.4f} {auc:>8.4f}")
print("─" * 68)
# === Section 9.8 (cont.): Confusion matrices ===
fig, axes = plt.subplots(1, 3, figsize=(16, 4))
for ax, (name, (preds, _)) in zip(axes, shop_preds.items()):
ConfusionMatrixDisplay.from_predictions(
ys_test, preds, ax=ax, cmap="Blues",
display_labels=["No Rev.", "Revenue"])
ax.set_title(name, fontsize=12)
fig.suptitle("Confusion Matrices — Online Shoppers",
fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
# === Section 9.8 (cont.): ROC curves ===
fig, ax = plt.subplots(figsize=(7, 6))
colours = ["steelblue", "#ef8a62", "#b2df8a"]
for (name, (_, proba)), colour in zip(shop_preds.items(), colours):
fpr, tpr, _ = roc_curve(ys_test, proba)
auc = roc_auc_score(ys_test, proba)
ax.plot(fpr, tpr, color=colour, linewidth=2,
label=f"{name} (AUC = {auc:.3f})")
ax.plot([0, 1], [0, 1], "--", color="gray", linewidth=1)
ax.set_xlabel("False Positive Rate")
ax.set_ylabel("True Positive Rate")
ax.set_title("ROC Curves — Online Shoppers", fontsize=13)
ax.legend(loc="lower right")
plt.tight_layout()
plt.show()
# === Section 9.8 (cont.): Feature importance — XGBoost ===
xgb_shop = shop_models["XGBoost"]
imp_shop = pd.Series(xgb_shop.feature_importances_, index=X_shop.columns)
top15 = imp_shop.nlargest(15).sort_values(ascending=True)
fig, ax = plt.subplots(figsize=(8, 6))
top15.plot.barh(ax=ax, color="steelblue", edgecolor="white")
ax.set_xlabel("XGBoost Gain Importance")
ax.set_title("Top 15 Features — Online Shopper Revenue", fontsize=13)
plt.tight_layout()
plt.show()
print(f"→ Top predictor: {imp_shop.idxmax()} "
f"(importance = {imp_shop.max():.4f}).")
# === Section 9.8 (cont.): Key findings & business recommendations ===
print("═══ Online Shopper Revenue — Key Findings ═══\n")
print("1. All three ensemble methods achieve strong AUC despite")
print(" significant class imbalance (only 15% of sessions convert).\n")
print("2. PageValues is the dominant predictor across all models —")
print(" sessions where the visitor sees product/cart pages are")
print(" far more likely to convert.\n")
print("3. BounceRates and ExitRates are the next most important,")
print(" confirming that engagement metrics predict conversion.\n")
print("4. Month and VisitorType provide modest signal — returning")
print(" visitors and November sessions convert at higher rates.\n")
print("5. Business recommendations:")
print(" • Prioritise UX improvements on product pages (PageValues).")
print(" • Target re-engagement campaigns at returning visitors.")
print(" • Investigate high-bounce pages for friction reduction.")
print(" • Deploy the XGBoost model for real-time conversion scoring.")
Section 9.9 — Summary & Quick Reference
Key Formulas¶
| Concept | Formula |
|---|---|
| Bias–Variance | $\text{Error} = \text{Bias}^2 + \text{Variance} + \text{Noise}$ |
| Bagging prediction | $\hat{y}_{\text{bag}} = \text{mode}\{\hat{y}_1, \ldots, \hat{y}_T\}$ |
| AdaBoost weight | $\alpha_t = \tfrac{1}{2}\ln\tfrac{1-\varepsilon_t}{\varepsilon_t}$ |
| Gradient Boosting update | $F_t = F_{t-1} + \eta\, h_t$ |
| XGBoost objective | $\mathcal{L} = \sum l(y_i,\hat{y}_i) + \sum [\gamma T_t + \tfrac{1}{2}\lambda\|w_t\|^2]$ |
Ensemble Decision Guide¶
| Method | Strengths | Best For |
|---|---|---|
| Bagging / RF | Reduces variance, parallel training, OOB score | High-variance models, quick baseline |
| AdaBoost | Simple, interpretable weights | Small datasets, weak learners |
| Gradient Boosting | Flexible loss functions, strong performance | Tabular data, competitions |
| XGBoost | Regularisation, speed, missing-value handling | Production, large datasets |
| Stacking | Combines diverse model strengths | Squeezing out final % of performance |
Python Cheat Sheet¶
from sklearn.ensemble import (
RandomForestClassifier, GradientBoostingClassifier,
StackingClassifier,
)
import xgboost as xgb
# Random Forest
rf = RandomForestClassifier(n_estimators=200, oob_score=True)
rf.fit(X_train, y_train)
print(rf.oob_score_)
# XGBoost
xgb_clf = xgb.XGBClassifier(n_estimators=200, learning_rate=0.1)
xgb_clf.fit(X_train, y_train)
Key Takeaways¶
$$\boxed{\text{Ensembles combine many weak learners into one strong learner — bagging reduces variance, boosting reduces bias.}}$$
- Bagging averages independent models to reduce variance.
- Random Forest adds feature sub-sampling to de-correlate trees.
- Boosting (AdaBoost, Gradient Boosting, XGBoost) trains sequentially, correcting prior errors.
- Stacking blends diverse models with a meta-learner.
- For tabular data, gradient boosting and XGBoost are the go-to methods.
- Always compare ensembles against a single-tree baseline to quantify the improvement.
Chapter 8 discovered structure without labels. Chapter 9 showed how combining models produces predictions far stronger than any single model.
Next → Chapter 10: Linear Algebra
References¶
Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5-32. The original random forest paper.
Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of KDD, 785-794. The paper introducing the gradient-boosting library used in this chapter.
Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29(5), 1189-1232. The theoretical foundation of gradient boosting.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. Chapter 9 (trees) and Chapter 10 (boosting).
scikit-learn Developers. scikit-learn
ensembledocumentation (RandomForest, GradientBoosting). https://scikit-learn.org/stable/modules/ensemble.html