Home / About Us / Volume I Contents / Ch 07: Classification and Model Selection

Statistical Basis of AI/ML — Classification \& Model Selection

Chapter 7 — Predicting Categorical Outcomes


Why This Chapter?

In Chapter 6 you built regression models to predict continuous outcomes. But many real-world questions are about categories: will the patient develop diabetes? Will the customer churn? Is the tumour malignant or benign?

Classification adapts the regression framework — using a sigmoid or softmax link function — to output probabilities over discrete classes. But predicting classes introduces new challenges: class imbalance, threshold selection, and metrics that go far beyond simple accuracy.

This chapter covers the four core classifiers (logistic regression, decision trees, K-nearest neighbours, support vector machines), the evaluation metrics that matter for classification, and the model-selection machinery (cross-validation, grid search, pipelines) that makes your workflow reproducible and rigorous.

What You Will Learn

$\S$ Topic Description
7.0 Setup Imports, GPU check, plot configuration
7.1 Logistic Regression Sigmoid function, binary classification, decision boundary
7.2 Decision Trees CART algorithm, Gini impurity, tree visualisation, feature importance
7.3 K-Nearest Neighbours Distance metrics, effect of $k$, scaling sensitivity
7.4 Evaluation Metrics Confusion matrix, precision, recall, F1, ROC-AUC, PR curve
7.5 Cross-Validation & Tuning K-fold, stratified, GridSearchCV, RandomizedSearchCV
7.6 Model Comparison Pipeline sklearn.pipeline, head-to-head comparison table
7.7 Case Study — Bank Customer Churn Full EDA → model → evaluation → business recommendations
7.8 Summary & Quick Reference Formulas, decision guide, cheat sheet

Section 7.0 — Setup: Imports & Configuration

This chapter adds classification-specific tools from scikit-learn.

Library Role in This Chapter
NumPy Array arithmetic, random seeding
Pandas DataFrames, data wrangling
Matplotlib Decision boundaries, ROC curves
Seaborn Confusion-matrix heat maps, pair plots
scikit-learn Classifiers, metrics, model selection, pipelines
PyTorch GPU availability check
In [1]:
# === Chapter 7: Setup ===

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import scipy
from pathlib import Path

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

from sklearn.model_selection import (
    train_test_split, cross_val_score, StratifiedKFold,
    GridSearchCV, RandomizedSearchCV,
)
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.pipeline import Pipeline
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, classification_report,
    roc_auc_score, roc_curve, precision_recall_curve,
    average_precision_score, ConfusionMatrixDisplay,
)
import sklearn

# ── Plot defaults (house style) ──
plt.rcParams.update({
    "figure.figsize": (8, 4),
    "axes.grid": True,
    "grid.alpha": 0.3,
    "axes.spines.top": False,
    "axes.spines.right": False,
})
sns.set_theme(style="whitegrid")

rng = np.random.default_rng(42)

DATA = Path("../../data")

print(f"NumPy {np.__version__}  |  Pandas {pd.__version__}  |  "
      f"SciPy {scipy.__version__}")
print(f"scikit-learn {sklearn.__version__}")
print("\nChapter 7 environment ready.")
NumPy 2.5.1  |  Pandas 2.3.3  |  SciPy 1.18.0
scikit-learn 1.9.0

Chapter 7 environment ready.

Section 7.1 — Logistic Regression

From Regression to Classification

Linear regression predicts any real number, but a probability must lie in $[0, 1]$. Logistic regression wraps the linear predictor in the sigmoid (logistic) function:

$$\boxed{\hat{p} = \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad z = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p}$$

The model predicts class 1 when $\hat{p} \ge 0.5$ (by default) and class 0 otherwise.

The Log-Odds (Logit) Interpretation

$$\boxed{\log\frac{\hat{p}}{1 - \hat{p}} = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p}$$

Each $\beta_j$ is the change in log-odds per unit increase in $x_j$. Exponentiating gives the odds ratio: $e^{\beta_j}$.

We use the Wisconsin Breast Cancer dataset — 569 tumours described by 30 features derived from cell-nucleus images. Target: diagnosis (M = malignant, B = benign).

In [2]:
# === Section 7.1: Load Wisconsin Breast Cancer & logistic regression ===

df_bc = pd.read_csv(DATA / "wisc_bc_data.csv")
df_bc["diagnosis"] = LabelEncoder().fit_transform(df_bc["diagnosis"])  # B=0, M=1
print(f"Wisconsin Breast Cancer: {df_bc.shape[0]} rows × {df_bc.shape[1]} columns")
print(f"Class balance: {dict(df_bc['diagnosis'].value_counts())}")

# Use two features for visualisation, all features for modelling later
feat2 = ["mean radius", "mean texture"]
X2 = df_bc[feat2].values
y_bc = df_bc["diagnosis"].values

# Train/test split
X2_train, X2_test, y2_train, y2_test = train_test_split(
    X2, y_bc, test_size=0.30, random_state=42, stratify=y_bc
)

# Fit logistic regression (scaled)
scaler2 = StandardScaler().fit(X2_train)
lr2 = LogisticRegression(random_state=42).fit(scaler2.transform(X2_train), y2_train)

acc_train = lr2.score(scaler2.transform(X2_train), y2_train)
acc_test  = lr2.score(scaler2.transform(X2_test), y2_test)
print(f"\nLogistic Regression (2 features):")
print(f"  Train accuracy = {acc_train:.4f}")
print(f"  Test  accuracy = {acc_test:.4f}")
Wisconsin Breast Cancer: 569 rows × 31 columns
Class balance: {0: np.int64(357), 1: np.int64(212)}

Logistic Regression (2 features):
  Train accuracy = 0.8844
  Test  accuracy = 0.8947
In [3]:
# === Section 7.1: Sigmoid curve & 2-D decision boundary ===

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# 1. Sigmoid function
z = np.linspace(-7, 7, 300)
axes[0].plot(z, 1 / (1 + np.exp(-z)), color="steelblue", lw=2.5)
axes[0].axhline(0.5, color="red", ls="--", lw=1, label="Threshold = 0.5")
axes[0].axvline(0, color="grey", ls=":", lw=0.8)
axes[0].set_xlabel("z (linear predictor)")
axes[0].set_ylabel("σ(z)")
axes[0].set_title("The Sigmoid Function")
axes[0].legend()

# 2. Decision boundary
X2s = scaler2.transform(X2)
h = 0.05
x_min, x_max = X2s[:, 0].min() - 1, X2s[:, 0].max() + 1
y_min, y_max = X2s[:, 1].min() - 1, X2s[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                      np.arange(y_min, y_max, h))
Z = lr2.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

axes[1].contourf(xx, yy, Z, alpha=0.25, cmap="RdBu_r")
axes[1].scatter(X2s[y_bc == 0, 0], X2s[y_bc == 0, 1], s=15, alpha=0.6,
                color="steelblue", label="Benign")
axes[1].scatter(X2s[y_bc == 1, 0], X2s[y_bc == 1, 1], s=15, alpha=0.6,
                color="#ef8a62", label="Malignant")
axes[1].set_xlabel(f"{feat2[0]} (scaled)")
axes[1].set_ylabel(f"{feat2[1]} (scaled)")
axes[1].set_title("Logistic Regression — Decision Boundary")
axes[1].legend()

fig.suptitle("Logistic Regression Fundamentals", fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
No description has been provided for this image

Section 7.2 — Decision Trees

The CART Algorithm

A decision tree recursively splits the feature space using axis-aligned boundaries. At each node the algorithm chooses the feature and threshold that produce the purest child nodes.

Gini Impurity

$$\boxed{G = 1 - \sum_{k=1}^{K} p_k^2}$$

where $p_k$ is the proportion of class $k$ in a node. $G = 0$ means the node is pure (all one class).

Strengths & Weaknesses

Strength Weakness
Highly interpretable Prone to overfitting
Handles mixed feature types Unstable (small data changes → different tree)
No scaling needed Greedy; may miss global optimum
Captures non-linear relationships High variance

Decision trees are the foundation of ensemble methods (Random Forest, Gradient Boosting) covered in later chapters.

In [4]:
# === Section 7.2: Decision tree on Wisconsin BC (all features) ===

features_bc = [c for c in df_bc.columns if c != "diagnosis"]
X_bc = df_bc[features_bc].values

X_bc_train, X_bc_test, y_bc_train, y_bc_test = train_test_split(
    X_bc, y_bc, test_size=0.30, random_state=42, stratify=y_bc
)

# Fit with max_depth to avoid overfitting
dt = DecisionTreeClassifier(max_depth=4, random_state=42)
dt.fit(X_bc_train, y_bc_train)

print(f"Decision Tree (depth ≤ 4):")
print(f"  Train accuracy = {dt.score(X_bc_train, y_bc_train):.4f}")
print(f"  Test  accuracy = {dt.score(X_bc_test, y_bc_test):.4f}")

# Visualise the tree
fig, ax = plt.subplots(figsize=(18, 8))
plot_tree(dt, feature_names=features_bc, class_names=["Benign", "Malignant"],
          filled=True, rounded=True, ax=ax, fontsize=8)
ax.set_title("Decision Tree — Wisconsin Breast Cancer (max_depth=4)",
             fontsize=14)
plt.tight_layout()
plt.show()
Decision Tree (depth ≤ 4):
  Train accuracy = 0.9774
  Test  accuracy = 0.9064
No description has been provided for this image
In [5]:
# === Section 7.2 (cont.): Feature importance ===

importance = pd.Series(dt.feature_importances_, index=features_bc)
top10 = importance.nlargest(10)

fig, ax = plt.subplots(figsize=(8, 5))
top10.sort_values().plot(kind="barh", ax=ax, color="steelblue",
                         edgecolor="white", linewidth=0.5)
ax.set_xlabel("Gini Importance")
ax.set_title("Top 10 Feature Importances — Decision Tree")
plt.tight_layout()
plt.show()

print("→ 'worst' features (computed from the three largest nuclei) dominate,")
print("  which aligns with the clinical intuition that the most abnormal cells")
print("  are the strongest predictors of malignancy.")
No description has been provided for this image
→ 'worst' features (computed from the three largest nuclei) dominate,
  which aligns with the clinical intuition that the most abnormal cells
  are the strongest predictors of malignancy.

Section 7.3 — K-Nearest Neighbours (KNN)

The Algorithm

KNN is a lazy learner — it stores the training data and classifies a new point by majority vote among its $k$ nearest neighbours.

Distance Metric

$$\boxed{d(\mathbf{a}, \mathbf{b}) = \sqrt{\sum_{j=1}^{p}(a_j - b_j)^2}}$$

(Euclidean by default; Manhattan $\sum|a_j - b_j|$ is another option.)

Key Considerations

  • Feature scaling is critical — KNN is distance-based, so unscaled features with large ranges dominate.
  • $k$ controls the bias–variance trade-off: small $k$ = low bias / high variance; large $k$ = high bias / low variance.
  • Curse of dimensionality: performance degrades in very high dimensions.
In [6]:
# === Section 7.3: KNN — effect of k on accuracy ===

scaler_bc = StandardScaler().fit(X_bc_train)
X_bc_train_s = scaler_bc.transform(X_bc_train)
X_bc_test_s  = scaler_bc.transform(X_bc_test)

k_values = list(range(1, 31))
train_acc = []
test_acc = []

for k in k_values:
    knn = KNeighborsClassifier(n_neighbors=k).fit(X_bc_train_s, y_bc_train)
    train_acc.append(knn.score(X_bc_train_s, y_bc_train))
    test_acc.append(knn.score(X_bc_test_s, y_bc_test))

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(k_values, train_acc, "o-", color="steelblue", ms=4, label="Train")
ax.plot(k_values, test_acc, "o-", color="#ef8a62", ms=4, label="Test")
best_k = k_values[np.argmax(test_acc)]
ax.axvline(best_k, color="green", ls="--", lw=1,
           label=f"Best k = {best_k} (test acc = {max(test_acc):.4f})")
ax.set_xlabel("k (number of neighbours)")
ax.set_ylabel("Accuracy")
ax.set_title("KNN — Effect of k on Classification Accuracy")
ax.legend()
plt.tight_layout()
plt.show()

print(f"→ Best test accuracy = {max(test_acc):.4f} at k = {best_k}")
print("→ k = 1 overfits (train = 1.0); large k smooths the boundary.")
No description has been provided for this image
→ Best test accuracy = 0.9649 at k = 5
→ k = 1 overfits (train = 1.0); large k smooths the boundary.

Section 7.4 — Evaluation Metrics: Beyond Accuracy

Accuracy alone is misleading with imbalanced classes. A model that always predicts "benign" on 63 % benign data gets 63 % accuracy while missing every cancer case.

The Confusion Matrix

Predicted Positive Predicted Negative
Actual Positive TP FN
Actual Negative FP TN

Key Metrics

$$\boxed{\text{Precision} = \frac{TP}{TP + FP}} \qquad \boxed{\text{Recall} = \frac{TP}{TP + FN}}$$

$$\boxed{F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}}$$

$$\boxed{\text{ROC-AUC} = \text{Area under the ROC curve (TPR vs. FPR)}}$$

Metric Best When
Precision Cost of FP is high (e.g., spam filter)
Recall Cost of FN is high (e.g., cancer screening)
F1 You need a single balanced metric
ROC-AUC Comparing models across all thresholds
In [7]:
# === Section 7.4: Confusion matrix & classification report ===

# Use logistic regression on ALL features for evaluation demo
lr_full = LogisticRegression(max_iter=5000, random_state=42)
lr_full.fit(X_bc_train_s, y_bc_train)
y_pred_lr = lr_full.predict(X_bc_test_s)
y_prob_lr = lr_full.predict_proba(X_bc_test_s)[:, 1]

fig, axes = plt.subplots(1, 2, figsize=(12, 4.5))

# 1. Confusion matrix
cm = confusion_matrix(y_bc_test, y_pred_lr)
ConfusionMatrixDisplay(cm, display_labels=["Benign", "Malignant"]).plot(
    ax=axes[0], cmap="Blues", colorbar=False)
axes[0].set_title("Confusion Matrix — Logistic Regression")

# 2. Normalised confusion matrix
cm_norm = cm.astype(float) / cm.sum(axis=1, keepdims=True)
ConfusionMatrixDisplay(cm_norm, display_labels=["Benign", "Malignant"]).plot(
    ax=axes[1], cmap="Blues", colorbar=False, values_format=".2%")
axes[1].set_title("Normalised Confusion Matrix")

plt.tight_layout()
plt.show()

print(classification_report(y_bc_test, y_pred_lr,
                            target_names=["Benign", "Malignant"]))
No description has been provided for this image
              precision    recall  f1-score   support

      Benign       0.96      0.99      0.98       107
   Malignant       0.98      0.94      0.96        64

    accuracy                           0.97       171
   macro avg       0.97      0.96      0.97       171
weighted avg       0.97      0.97      0.97       171

In [8]:
# === Section 7.4 (cont.): ROC curve & Precision–Recall curve ===

fpr, tpr, _ = roc_curve(y_bc_test, y_prob_lr)
auc_val = roc_auc_score(y_bc_test, y_prob_lr)

prec_curve, rec_curve, _ = precision_recall_curve(y_bc_test, y_prob_lr)
ap = average_precision_score(y_bc_test, y_prob_lr)

fig, axes = plt.subplots(1, 2, figsize=(13, 5))

# ROC
axes[0].plot(fpr, tpr, color="steelblue", lw=2,
             label=f"Logistic Reg (AUC = {auc_val:.3f})")
axes[0].plot([0, 1], [0, 1], "r--", lw=1, label="Random (AUC = 0.5)")
axes[0].set_xlabel("False Positive Rate")
axes[0].set_ylabel("True Positive Rate")
axes[0].set_title("ROC Curve")
axes[0].legend(loc="lower right")

# PR
axes[1].plot(rec_curve, prec_curve, color="#ef8a62", lw=2,
             label=f"Logistic Reg (AP = {ap:.3f})")
axes[1].set_xlabel("Recall")
axes[1].set_ylabel("Precision")
axes[1].set_title("Precision–Recall Curve")
axes[1].legend(loc="lower left")

fig.suptitle("Threshold-Independent Evaluation — Wisconsin BC",
             fontsize=14, y=1.02)
plt.tight_layout()
plt.show()

print(f"→ ROC-AUC = {auc_val:.4f}  |  Average Precision = {ap:.4f}")
print("→ Both curves show the model distinguishes classes very well.")
No description has been provided for this image
→ ROC-AUC = 0.9975  |  Average Precision = 0.9963
→ Both curves show the model distinguishes classes very well.

Section 7.5 — Cross-Validation & Hyperparameter Tuning

Why Cross-Validation?

A single train/test split is sensitive to the random split. K-fold cross-validation partitions the data into $k$ folds and trains $k$ models, each time holding out a different fold for validation.

$$\boxed{\text{CV Score} = \frac{1}{k}\sum_{i=1}^{k} \text{Score}_i}$$

Stratified K-Fold

For classification, stratified K-fold ensures each fold preserves the class distribution — essential with imbalanced data.

GridSearchCV & RandomizedSearchCV

  • GridSearchCV: exhaustive search over a parameter grid.
  • RandomizedSearchCV: samples a fixed number of random combinations — faster when the grid is large.

Both use cross-validation internally to estimate each configuration's performance.

In [9]:
# === Section 7.5: Stratified K-fold CV comparison ===

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

classifiers = {
    "Logistic Regression": LogisticRegression(max_iter=5000, random_state=42),
    "Decision Tree (d=4)": DecisionTreeClassifier(max_depth=4, random_state=42),
    "KNN (k=5)":           KNeighborsClassifier(n_neighbors=5),
    "SVM (RBF)":           SVC(random_state=42),
}

print("═══ 5-Fold Stratified CV — Wisconsin BC (scaled) ═══\n")
print(f"{'Classifier':25s}  {'Mean Acc':>9s}  {'Std':>7s}  {'Scores'}")
print("─" * 75)

cv_results = {}
for name, clf in classifiers.items():
    pipe = Pipeline([("scaler", StandardScaler()), ("clf", clf)])
    scores = cross_val_score(pipe, X_bc, y_bc, cv=cv, scoring="accuracy")
    cv_results[name] = scores
    print(f"  {name:25s}  {scores.mean():9.4f}  {scores.std():7.4f}  "
          f"{np.round(scores, 4)}")

print("\n→ All classifiers perform well on this dataset (clear class separation).")
═══ 5-Fold Stratified CV — Wisconsin BC (scaled) ═══

Classifier                  Mean Acc      Std  Scores
───────────────────────────────────────────────────────────────────────────
  Logistic Regression           0.9737   0.0166  [0.9737 0.9474 0.9649 0.9912 0.9912]
  Decision Tree (d=4)           0.9227   0.0295  [0.9561 0.8684 0.9211 0.9298 0.9381]
  KNN (k=5)                     0.9631   0.0179  [0.9825 0.9474 0.9386 0.9825 0.9646]
  SVM (RBF)                     0.9772   0.0163  [0.9912 0.9474 0.9737 0.9912 0.9823]

→ All classifiers perform well on this dataset (clear class separation).
In [10]:
# === Section 7.5 (cont.): GridSearchCV for KNN ===

pipe_knn = Pipeline([
    ("scaler", StandardScaler()),
    ("knn", KNeighborsClassifier()),
])

param_grid = {
    "knn__n_neighbors": [3, 5, 7, 9, 11, 15, 21],
    "knn__weights": ["uniform", "distance"],
    "knn__metric": ["euclidean", "manhattan"],
}

grid = GridSearchCV(pipe_knn, param_grid, cv=cv, scoring="accuracy",
                    n_jobs=-1, return_train_score=True)
grid.fit(X_bc, y_bc)

print("═══ GridSearchCV — KNN ═══\n")
print(f"  Best params:     {grid.best_params_}")
print(f"  Best CV accuracy: {grid.best_score_:.4f}")
print(f"  Total configs tested: {len(grid.cv_results_['mean_test_score'])}")

# Top 5 configurations
results_df = pd.DataFrame(grid.cv_results_)
top5 = (results_df[["param_knn__n_neighbors", "param_knn__weights",
                     "param_knn__metric", "mean_test_score", "std_test_score"]]
         .sort_values("mean_test_score", ascending=False)
         .head(5))
print("\nTop 5 configurations:")
print(top5.to_string(index=False))
═══ GridSearchCV — KNN ═══

  Best params:     {'knn__metric': 'euclidean', 'knn__n_neighbors': 9, 'knn__weights': 'distance'}
  Best CV accuracy: 0.9684
  Total configs tested: 28

Top 5 configurations:
 param_knn__n_neighbors param_knn__weights param_knn__metric  mean_test_score  std_test_score
                      9           distance         euclidean         0.968374        0.014236
                      9           distance         manhattan         0.968359        0.016283
                      9            uniform         manhattan         0.968359        0.016283
                      7           distance         euclidean         0.966620        0.016059
                      9            uniform         euclidean         0.966620        0.016059

Section 7.6 — Model Comparison Pipeline

sklearn.pipeline.Pipeline chains preprocessing and modelling into a single estimator that is safe to use inside cross-validation (no data leakage from the scaler).

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression()),
])
cross_val_score(pipe, X, y, cv=5)  # scaler fits only on train folds

We now run a head-to-head comparison of six classifiers using pipelines, plotting their CV score distributions.

In [11]:
# === Section 7.6: Head-to-head model comparison ===

# Strip 'knn__' prefix from GridSearchCV best params for standalone KNN
knn_params = {k.replace("knn__", ""): v for k, v in grid.best_params_.items()}

all_classifiers = {
    "Logistic Reg":      LogisticRegression(max_iter=5000, random_state=42),
    "Decision Tree":     DecisionTreeClassifier(max_depth=4, random_state=42),
    "KNN (best)":        KNeighborsClassifier(**knn_params),
    "SVM (RBF)":         SVC(random_state=42),
    "Random Forest":     RandomForestClassifier(n_estimators=100, random_state=42),
    "Gradient Boosting": GradientBoostingClassifier(n_estimators=100, random_state=42),
}

cv_all = {}
for name, clf in all_classifiers.items():
    pipe = Pipeline([("scaler", StandardScaler()), ("clf", clf)])
    scores = cross_val_score(pipe, X_bc, y_bc, cv=cv, scoring="accuracy")
    cv_all[name] = scores

# Summary table
print("═══ Model Comparison — 5-Fold Stratified CV ═══\n")
print(f"{'Classifier':20s}  {'Mean':>7s}  {'Std':>6s}  {'Min':>6s}  {'Max':>6s}")
print("─" * 50)
for name, scores in cv_all.items():
    print(f"  {name:20s}  {scores.mean():7.4f}  {scores.std():6.4f}  "
          f"{scores.min():6.4f}  {scores.max():6.4f}")
═══ Model Comparison — 5-Fold Stratified CV ═══

Classifier               Mean     Std     Min     Max
──────────────────────────────────────────────────
  Logistic Reg           0.9737  0.0166  0.9474  0.9912
  Decision Tree          0.9227  0.0295  0.8684  0.9561
  KNN (best)             0.9684  0.0142  0.9474  0.9825
  SVM (RBF)              0.9772  0.0163  0.9474  0.9912
  Random Forest          0.9543  0.0128  0.9386  0.9735
  Gradient Boosting      0.9508  0.0245  0.9035  0.9735
In [12]:
# === Section 7.6 (cont.): CV score distribution box plot ===

fig, ax = plt.subplots(figsize=(10, 5))
cv_df = pd.DataFrame(cv_all)
cv_df_melted = cv_df.melt(var_name="Classifier", value_name="Accuracy")

sns.boxplot(data=cv_df_melted, x="Classifier", y="Accuracy", ax=ax,
            palette=["steelblue", "#67a9cf", "#ef8a62", "#b2df8a",
                     "#2166ac", "#d1e5f0"])
ax.set_ylim(0.90, 1.005)
ax.set_title("Cross-Validation Accuracy Distribution — Wisconsin BC")
ax.set_xlabel("")
plt.xticks(rotation=20, ha="right")
plt.tight_layout()
plt.show()

best_model = max(cv_all, key=lambda k: cv_all[k].mean())
print(f"→ Best overall: {best_model} "
      f"(mean CV accuracy = {cv_all[best_model].mean():.4f})")
/tmp/ipykernel_704/1455900797.py:7: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(data=cv_df_melted, x="Classifier", y="Accuracy", ax=ax,
No description has been provided for this image
→ Best overall: SVM (RBF) (mean CV accuracy = 0.9772)

Section 7.7 — Case Study: Bank Customer Churn

We now apply the full classification workflow to the Bank Churners dataset — 10,127 credit-card customers with 21 attributes covering demographics, transaction behaviour, and account metrics.

Target: Attrition_Flag (Existing Customer vs. Attrited Customer)

This is a realistic imbalanced dataset (≈ 84 % existing vs. 16 % attrited), which makes it ideal for practising the evaluation techniques from §7.4.

Workflow:

  1. EDA — distributions, class balance, correlations
  2. Data Preparation — encoding, scaling, train/test split
  3. Model Fitting — logistic regression, decision tree, random forest
  4. Evaluation — confusion matrix, ROC-AUC, PR curve
  5. Business Recommendations
In [13]:
# === Section 7.7: Load Bank Churn & EDA ===

df_churn = pd.read_csv(DATA / "bankchurn.csv")
print(f"Bank Churn: {df_churn.shape[0]} rows × {df_churn.shape[1]} columns")
print(f"\nTarget distribution:")
print(df_churn["Attrition_Flag"].value_counts())
print(f"\nImbalance ratio: "
      f"{df_churn['Attrition_Flag'].value_counts(normalize=True).iloc[0]:.1%} / "
      f"{df_churn['Attrition_Flag'].value_counts(normalize=True).iloc[1]:.1%}")

fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))

# 1. Target balance
df_churn["Attrition_Flag"].value_counts().plot(kind="bar", ax=axes[0],
    color=["steelblue", "#ef8a62"], edgecolor="white")
axes[0].set_title("Class Distribution")
axes[0].set_ylabel("Count")
axes[0].set_xticklabels(axes[0].get_xticklabels(), rotation=0)

# 2. Age distribution by class
for label, color in zip(["Existing Customer", "Attrited Customer"],
                         ["steelblue", "#ef8a62"]):
    axes[1].hist(df_churn.loc[df_churn["Attrition_Flag"] == label, "Customer_Age"],
                 bins=30, alpha=0.5, color=color, label=label, edgecolor="white")
axes[1].set_xlabel("Customer Age")
axes[1].set_title("Age Distribution by Churn Status")
axes[1].legend(fontsize=8)

# 3. Transaction count vs. total amount
for label, color in zip(["Existing Customer", "Attrited Customer"],
                         ["steelblue", "#ef8a62"]):
    mask = df_churn["Attrition_Flag"] == label
    axes[2].scatter(df_churn.loc[mask, "Total_Trans_Ct"],
                    df_churn.loc[mask, "Total_Trans_Amt"],
                    s=5, alpha=0.3, color=color, label=label)
axes[2].set_xlabel("Total Transaction Count")
axes[2].set_ylabel("Total Transaction Amount")
axes[2].set_title("Transactions by Churn Status")
axes[2].legend(fontsize=8)

fig.suptitle("Bank Customer Churn — Exploratory Analysis", fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
Bank Churn: 10127 rows × 21 columns

Target distribution:
Attrition_Flag
Existing Customer    8500
Attrited Customer    1627
Name: count, dtype: int64

Imbalance ratio: 83.9% / 16.1%
No description has been provided for this image
In [14]:
# === Section 7.7: Data preparation ===

df_ch = df_churn.copy()

# Encode target: 1 = Attrited, 0 = Existing
df_ch["Churn"] = (df_ch["Attrition_Flag"] == "Attrited Customer").astype(int)
df_ch.drop(columns=["Attrition_Flag", "CLIENTNUM"], inplace=True)

# One-hot encode categorical columns
cat_cols = df_ch.select_dtypes(include="object").columns.tolist()
df_ch = pd.get_dummies(df_ch, columns=cat_cols, dtype=int)

# Features & target
target_ch = "Churn"
features_ch = [c for c in df_ch.columns if c != target_ch]
X_ch = df_ch[features_ch].values
y_ch = df_ch[target_ch].values

X_ch_train, X_ch_test, y_ch_train, y_ch_test = train_test_split(
    X_ch, y_ch, test_size=0.30, random_state=42, stratify=y_ch
)

scaler_ch = StandardScaler().fit(X_ch_train)
X_ch_train_s = scaler_ch.transform(X_ch_train)
X_ch_test_s  = scaler_ch.transform(X_ch_test)

print(f"Features: {len(features_ch)}")
print(f"Training set: {X_ch_train_s.shape}  |  Test set: {X_ch_test_s.shape}")
print(f"Train class balance: {dict(pd.Series(y_ch_train).value_counts())}")
Features: 37
Training set: (7088, 37)  |  Test set: (3039, 37)
Train class balance: {0: np.int64(5949), 1: np.int64(1139)}
In [15]:
# === Section 7.7: Fit three classifiers ===

models_ch = {
    "Logistic Reg": LogisticRegression(max_iter=5000, random_state=42),
    "Decision Tree": DecisionTreeClassifier(max_depth=5, random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=200, random_state=42),
}

predictions_ch = {}
probabilities_ch = {}

print("═══ Bank Churn — Model Training ═══\n")
for name, clf in models_ch.items():
    clf.fit(X_ch_train_s, y_ch_train)
    y_pred = clf.predict(X_ch_test_s)
    y_prob = clf.predict_proba(X_ch_test_s)[:, 1]
    predictions_ch[name] = y_pred
    probabilities_ch[name] = y_prob

    acc = accuracy_score(y_ch_test, y_pred)
    prec = precision_score(y_ch_test, y_pred)
    rec = recall_score(y_ch_test, y_pred)
    f1 = f1_score(y_ch_test, y_pred)
    auc = roc_auc_score(y_ch_test, y_prob)
    print(f"  {name:16s}  Acc={acc:.4f}  Prec={prec:.4f}  "
          f"Rec={rec:.4f}  F1={f1:.4f}  AUC={auc:.4f}")
═══ Bank Churn — Model Training ═══

  Logistic Reg      Acc=0.8990  Prec=0.7638  Rec=0.5369  F1=0.6306  AUC=0.9145
  Decision Tree     Acc=0.9283  Prec=0.7679  Rec=0.7930  F1=0.7802  AUC=0.9421
  Random Forest     Acc=0.9526  Prec=0.9175  Rec=0.7746  F1=0.8400  AUC=0.9856
In [16]:
# === Section 7.7: Confusion matrices for all models ===

fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))

for ax, (name, y_pred) in zip(axes, predictions_ch.items()):
    cm = confusion_matrix(y_ch_test, y_pred)
    ConfusionMatrixDisplay(cm, display_labels=["Existing", "Attrited"]).plot(
        ax=ax, cmap="Blues", colorbar=False)
    ax.set_title(name)

fig.suptitle("Confusion Matrices — Bank Customer Churn", fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
No description has been provided for this image
In [17]:
# === Section 7.7: ROC & PR curves — all models ===

colors = ["steelblue", "#ef8a62", "#b2df8a"]

fig, axes = plt.subplots(1, 2, figsize=(13, 5))

for (name, y_prob), color in zip(probabilities_ch.items(), colors):
    # ROC
    fpr, tpr, _ = roc_curve(y_ch_test, y_prob)
    auc_val = roc_auc_score(y_ch_test, y_prob)
    axes[0].plot(fpr, tpr, color=color, lw=2,
                 label=f"{name} (AUC={auc_val:.3f})")
    # PR
    prec_c, rec_c, _ = precision_recall_curve(y_ch_test, y_prob)
    ap = average_precision_score(y_ch_test, y_prob)
    axes[1].plot(rec_c, prec_c, color=color, lw=2,
                 label=f"{name} (AP={ap:.3f})")

axes[0].plot([0, 1], [0, 1], "r--", lw=1)
axes[0].set_xlabel("False Positive Rate")
axes[0].set_ylabel("True Positive Rate")
axes[0].set_title("ROC Curves")
axes[0].legend(loc="lower right", fontsize=9)

axes[1].set_xlabel("Recall")
axes[1].set_ylabel("Precision")
axes[1].set_title("Precision–Recall Curves")
axes[1].legend(loc="lower left", fontsize=9)

fig.suptitle("Model Comparison — Bank Customer Churn", fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
No description has been provided for this image
In [18]:
# === Section 7.7: Feature importance — Random Forest ===

rf_imp = pd.Series(
    models_ch["Random Forest"].feature_importances_, index=features_ch
)
top15 = rf_imp.nlargest(15)

fig, ax = plt.subplots(figsize=(9, 6))
top15.sort_values().plot(kind="barh", ax=ax, color="steelblue",
                         edgecolor="white", linewidth=0.5)
ax.set_xlabel("Feature Importance (Gini)")
ax.set_title("Top 15 Features — Random Forest (Bank Churn)")
plt.tight_layout()
plt.show()

print("→ Transaction count & amount are the strongest churn predictors,")
print("  followed by revolving balance and relationship count.")
No description has been provided for this image
→ Transaction count & amount are the strongest churn predictors,
  followed by revolving balance and relationship count.
In [19]:
# === Section 7.7: Key findings & business recommendations ===

print("═══ Key Findings — Bank Customer Churn ═══\n")

best_name = max(probabilities_ch,
                key=lambda k: roc_auc_score(y_ch_test, probabilities_ch[k]))
best_auc = roc_auc_score(y_ch_test, probabilities_ch[best_name])
best_f1  = f1_score(y_ch_test, predictions_ch[best_name])

print(f"  Best model: {best_name}")
print(f"  ROC-AUC   = {best_auc:.4f}")
print(f"  F1-score  = {best_f1:.4f}")

print(f"\n  Top 5 churn predictors:")
for feat, imp in rf_imp.nlargest(5).items():
    print(f"    → {feat}: importance = {imp:.4f}")

print("\n═══ Business Recommendations ═══")
print("→ Transaction frequency and total amount are the strongest")
print("  signals of churn — declining engagement predicts attrition.")
print("→ Target customers with dropping transaction counts for")
print("  proactive retention campaigns (rewards, personal outreach).")
print("→ Monitor revolving balance and credit utilisation as")
print("  secondary indicators of disengagement.")
print("→ The Random Forest / Gradient Boosting models provide the")
print("  best discrimination — deploy with a recall-optimised threshold")
print("  to catch more at-risk customers (accept slightly more FPs).")
═══ Key Findings — Bank Customer Churn ═══

  Best model: Random Forest
  ROC-AUC   = 0.9856
  F1-score  = 0.8400

  Top 5 churn predictors:
    → Total_Trans_Amt: importance = 0.1772
    → Total_Trans_Ct: importance = 0.1563
    → Total_Ct_Chng_Q4_Q1: importance = 0.1076
    → Total_Revolving_Bal: importance = 0.1032
    → Total_Relationship_Count: importance = 0.0675

═══ Business Recommendations ═══
→ Transaction frequency and total amount are the strongest
  signals of churn — declining engagement predicts attrition.
→ Target customers with dropping transaction counts for
  proactive retention campaigns (rewards, personal outreach).
→ Monitor revolving balance and credit utilisation as
  secondary indicators of disengagement.
→ The Random Forest / Gradient Boosting models provide the
  best discrimination — deploy with a recall-optimised threshold
  to catch more at-risk customers (accept slightly more FPs).

Section 7.8 — Summary & Quick Reference

Key Formulas

Quantity Formula
Sigmoid $\sigma(z) = \dfrac{1}{1 + e^{-z}}$
Log-odds $\log\dfrac{p}{1-p} = \mathbf{x}^T \boldsymbol{\beta}$
Gini impurity $G = 1 - \sum p_k^2$
Euclidean distance $d = \sqrt{\sum(a_j - b_j)^2}$
Precision $TP / (TP + FP)$
Recall $TP / (TP + FN)$
$F_1$ score $2 \cdot \dfrac{P \cdot R}{P + R}$
K-fold CV $\dfrac{1}{k}\sum \text{Score}_i$

Classifier Decision Guide

Situation Recommended Classifier
Baseline / interpretable coefficients Logistic Regression
Non-linear boundaries, few features Decision Tree / KNN
High-dimensional, clear margin SVM (RBF kernel)
Best accuracy, many features Random Forest / Gradient Boosting
Need probability calibration Logistic Regression or calibrated ensemble

Python Cheat Sheet

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, roc_auc_score

# Pipeline:   pipe = Pipeline([("scaler", StandardScaler()), ("clf", model)])
# CV:         cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
# Grid:       GridSearchCV(pipe, param_grid, cv=5).fit(X, y)
# Report:     classification_report(y_true, y_pred)
# ROC-AUC:    roc_auc_score(y_true, y_prob)

Key Takeaways

$$\boxed{\text{Accuracy is not enough — use precision, recall, F1, and ROC-AUC for classification.}}$$

  • Logistic regression is the classification analogue of linear regression — start here for interpretability and baseline performance.
  • Decision trees are intuitive but overfit easily; random forests and gradient boosting fix this with ensembling.
  • KNN is simple but scaling-sensitive and slow on large datasets.
  • Cross-validation and pipelines make your workflow rigorous and prevent data leakage.
  • The case study demonstrated that on imbalanced data, the model with the highest accuracy is not necessarily the best — ROC-AUC and F1 give a more honest picture.

Chapter 6 predicted how much; this chapter predicts which class. Together they cover the two pillars of supervised learning.


Next → Chapter 8: Unsupervised Learning


References

  1. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.). Springer. Chapter 4 (classification) and Chapter 5 (resampling/model selection). https://www.statlearning.com/

  2. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. Chapter 4 (classification) and Chapter 7 (model assessment and selection).

  3. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of IJCAI, 14(2), 1137-1145. The classical reference for k-fold cross-validation.

  4. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273-297. The original SVM paper.

  5. scikit-learn Developers. scikit-learn model_selection and metrics documentation (GridSearchCV, ROC/AUC, confusion matrix). https://scikit-learn.org/stable/modules/model_selection.html