Statistical Basis of AI/ML — Classification \& Model Selection
Chapter 7 — Predicting Categorical Outcomes
Why This Chapter?
In Chapter 6 you built regression models to predict continuous outcomes. But many real-world questions are about categories: will the patient develop diabetes? Will the customer churn? Is the tumour malignant or benign?
Classification adapts the regression framework — using a sigmoid or softmax link function — to output probabilities over discrete classes. But predicting classes introduces new challenges: class imbalance, threshold selection, and metrics that go far beyond simple accuracy.
This chapter covers the four core classifiers (logistic regression, decision trees, K-nearest neighbours, support vector machines), the evaluation metrics that matter for classification, and the model-selection machinery (cross-validation, grid search, pipelines) that makes your workflow reproducible and rigorous.
What You Will Learn
| $\S$ | Topic | Description |
|---|---|---|
| 7.0 | Setup | Imports, GPU check, plot configuration |
| 7.1 | Logistic Regression | Sigmoid function, binary classification, decision boundary |
| 7.2 | Decision Trees | CART algorithm, Gini impurity, tree visualisation, feature importance |
| 7.3 | K-Nearest Neighbours | Distance metrics, effect of $k$, scaling sensitivity |
| 7.4 | Evaluation Metrics | Confusion matrix, precision, recall, F1, ROC-AUC, PR curve |
| 7.5 | Cross-Validation & Tuning | K-fold, stratified, GridSearchCV, RandomizedSearchCV |
| 7.6 | Model Comparison Pipeline | sklearn.pipeline, head-to-head comparison table |
| 7.7 | Case Study — Bank Customer Churn | Full EDA → model → evaluation → business recommendations |
| 7.8 | Summary & Quick Reference | Formulas, decision guide, cheat sheet |
Section 7.0 — Setup: Imports & Configuration
This chapter adds classification-specific tools from scikit-learn.
| Library | Role in This Chapter |
|---|---|
| NumPy | Array arithmetic, random seeding |
| Pandas | DataFrames, data wrangling |
| Matplotlib | Decision boundaries, ROC curves |
| Seaborn | Confusion-matrix heat maps, pair plots |
| scikit-learn | Classifiers, metrics, model selection, pipelines |
| PyTorch | GPU availability check |
# === Chapter 7: Setup ===
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import scipy
from pathlib import Path
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.model_selection import (
train_test_split, cross_val_score, StratifiedKFold,
GridSearchCV, RandomizedSearchCV,
)
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.pipeline import Pipeline
from sklearn.metrics import (
accuracy_score, precision_score, recall_score, f1_score,
confusion_matrix, classification_report,
roc_auc_score, roc_curve, precision_recall_curve,
average_precision_score, ConfusionMatrixDisplay,
)
import sklearn
# ── Plot defaults (house style) ──
plt.rcParams.update({
"figure.figsize": (8, 4),
"axes.grid": True,
"grid.alpha": 0.3,
"axes.spines.top": False,
"axes.spines.right": False,
})
sns.set_theme(style="whitegrid")
rng = np.random.default_rng(42)
DATA = Path("../../data")
print(f"NumPy {np.__version__} | Pandas {pd.__version__} | "
f"SciPy {scipy.__version__}")
print(f"scikit-learn {sklearn.__version__}")
print("\nChapter 7 environment ready.")
Section 7.1 — Logistic Regression
From Regression to Classification¶
Linear regression predicts any real number, but a probability must lie in $[0, 1]$. Logistic regression wraps the linear predictor in the sigmoid (logistic) function:
$$\boxed{\hat{p} = \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad z = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p}$$
The model predicts class 1 when $\hat{p} \ge 0.5$ (by default) and class 0 otherwise.
The Log-Odds (Logit) Interpretation¶
$$\boxed{\log\frac{\hat{p}}{1 - \hat{p}} = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p}$$
Each $\beta_j$ is the change in log-odds per unit increase in $x_j$. Exponentiating gives the odds ratio: $e^{\beta_j}$.
We use the Wisconsin Breast Cancer dataset — 569 tumours described by
30 features derived from cell-nucleus images. Target: diagnosis
(M = malignant, B = benign).
# === Section 7.1: Load Wisconsin Breast Cancer & logistic regression ===
df_bc = pd.read_csv(DATA / "wisc_bc_data.csv")
df_bc["diagnosis"] = LabelEncoder().fit_transform(df_bc["diagnosis"]) # B=0, M=1
print(f"Wisconsin Breast Cancer: {df_bc.shape[0]} rows × {df_bc.shape[1]} columns")
print(f"Class balance: {dict(df_bc['diagnosis'].value_counts())}")
# Use two features for visualisation, all features for modelling later
feat2 = ["mean radius", "mean texture"]
X2 = df_bc[feat2].values
y_bc = df_bc["diagnosis"].values
# Train/test split
X2_train, X2_test, y2_train, y2_test = train_test_split(
X2, y_bc, test_size=0.30, random_state=42, stratify=y_bc
)
# Fit logistic regression (scaled)
scaler2 = StandardScaler().fit(X2_train)
lr2 = LogisticRegression(random_state=42).fit(scaler2.transform(X2_train), y2_train)
acc_train = lr2.score(scaler2.transform(X2_train), y2_train)
acc_test = lr2.score(scaler2.transform(X2_test), y2_test)
print(f"\nLogistic Regression (2 features):")
print(f" Train accuracy = {acc_train:.4f}")
print(f" Test accuracy = {acc_test:.4f}")
# === Section 7.1: Sigmoid curve & 2-D decision boundary ===
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# 1. Sigmoid function
z = np.linspace(-7, 7, 300)
axes[0].plot(z, 1 / (1 + np.exp(-z)), color="steelblue", lw=2.5)
axes[0].axhline(0.5, color="red", ls="--", lw=1, label="Threshold = 0.5")
axes[0].axvline(0, color="grey", ls=":", lw=0.8)
axes[0].set_xlabel("z (linear predictor)")
axes[0].set_ylabel("σ(z)")
axes[0].set_title("The Sigmoid Function")
axes[0].legend()
# 2. Decision boundary
X2s = scaler2.transform(X2)
h = 0.05
x_min, x_max = X2s[:, 0].min() - 1, X2s[:, 0].max() + 1
y_min, y_max = X2s[:, 1].min() - 1, X2s[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
np.arange(y_min, y_max, h))
Z = lr2.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
axes[1].contourf(xx, yy, Z, alpha=0.25, cmap="RdBu_r")
axes[1].scatter(X2s[y_bc == 0, 0], X2s[y_bc == 0, 1], s=15, alpha=0.6,
color="steelblue", label="Benign")
axes[1].scatter(X2s[y_bc == 1, 0], X2s[y_bc == 1, 1], s=15, alpha=0.6,
color="#ef8a62", label="Malignant")
axes[1].set_xlabel(f"{feat2[0]} (scaled)")
axes[1].set_ylabel(f"{feat2[1]} (scaled)")
axes[1].set_title("Logistic Regression — Decision Boundary")
axes[1].legend()
fig.suptitle("Logistic Regression Fundamentals", fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
Section 7.2 — Decision Trees
The CART Algorithm¶
A decision tree recursively splits the feature space using axis-aligned boundaries. At each node the algorithm chooses the feature and threshold that produce the purest child nodes.
Gini Impurity¶
$$\boxed{G = 1 - \sum_{k=1}^{K} p_k^2}$$
where $p_k$ is the proportion of class $k$ in a node. $G = 0$ means the node is pure (all one class).
Strengths & Weaknesses¶
| Strength | Weakness |
|---|---|
| Highly interpretable | Prone to overfitting |
| Handles mixed feature types | Unstable (small data changes → different tree) |
| No scaling needed | Greedy; may miss global optimum |
| Captures non-linear relationships | High variance |
Decision trees are the foundation of ensemble methods (Random Forest, Gradient Boosting) covered in later chapters.
# === Section 7.2: Decision tree on Wisconsin BC (all features) ===
features_bc = [c for c in df_bc.columns if c != "diagnosis"]
X_bc = df_bc[features_bc].values
X_bc_train, X_bc_test, y_bc_train, y_bc_test = train_test_split(
X_bc, y_bc, test_size=0.30, random_state=42, stratify=y_bc
)
# Fit with max_depth to avoid overfitting
dt = DecisionTreeClassifier(max_depth=4, random_state=42)
dt.fit(X_bc_train, y_bc_train)
print(f"Decision Tree (depth ≤ 4):")
print(f" Train accuracy = {dt.score(X_bc_train, y_bc_train):.4f}")
print(f" Test accuracy = {dt.score(X_bc_test, y_bc_test):.4f}")
# Visualise the tree
fig, ax = plt.subplots(figsize=(18, 8))
plot_tree(dt, feature_names=features_bc, class_names=["Benign", "Malignant"],
filled=True, rounded=True, ax=ax, fontsize=8)
ax.set_title("Decision Tree — Wisconsin Breast Cancer (max_depth=4)",
fontsize=14)
plt.tight_layout()
plt.show()
# === Section 7.2 (cont.): Feature importance ===
importance = pd.Series(dt.feature_importances_, index=features_bc)
top10 = importance.nlargest(10)
fig, ax = plt.subplots(figsize=(8, 5))
top10.sort_values().plot(kind="barh", ax=ax, color="steelblue",
edgecolor="white", linewidth=0.5)
ax.set_xlabel("Gini Importance")
ax.set_title("Top 10 Feature Importances — Decision Tree")
plt.tight_layout()
plt.show()
print("→ 'worst' features (computed from the three largest nuclei) dominate,")
print(" which aligns with the clinical intuition that the most abnormal cells")
print(" are the strongest predictors of malignancy.")
Section 7.3 — K-Nearest Neighbours (KNN)
The Algorithm¶
KNN is a lazy learner — it stores the training data and classifies a new point by majority vote among its $k$ nearest neighbours.
Distance Metric¶
$$\boxed{d(\mathbf{a}, \mathbf{b}) = \sqrt{\sum_{j=1}^{p}(a_j - b_j)^2}}$$
(Euclidean by default; Manhattan $\sum|a_j - b_j|$ is another option.)
Key Considerations¶
- Feature scaling is critical — KNN is distance-based, so unscaled features with large ranges dominate.
- $k$ controls the bias–variance trade-off: small $k$ = low bias / high variance; large $k$ = high bias / low variance.
- Curse of dimensionality: performance degrades in very high dimensions.
# === Section 7.3: KNN — effect of k on accuracy ===
scaler_bc = StandardScaler().fit(X_bc_train)
X_bc_train_s = scaler_bc.transform(X_bc_train)
X_bc_test_s = scaler_bc.transform(X_bc_test)
k_values = list(range(1, 31))
train_acc = []
test_acc = []
for k in k_values:
knn = KNeighborsClassifier(n_neighbors=k).fit(X_bc_train_s, y_bc_train)
train_acc.append(knn.score(X_bc_train_s, y_bc_train))
test_acc.append(knn.score(X_bc_test_s, y_bc_test))
fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(k_values, train_acc, "o-", color="steelblue", ms=4, label="Train")
ax.plot(k_values, test_acc, "o-", color="#ef8a62", ms=4, label="Test")
best_k = k_values[np.argmax(test_acc)]
ax.axvline(best_k, color="green", ls="--", lw=1,
label=f"Best k = {best_k} (test acc = {max(test_acc):.4f})")
ax.set_xlabel("k (number of neighbours)")
ax.set_ylabel("Accuracy")
ax.set_title("KNN — Effect of k on Classification Accuracy")
ax.legend()
plt.tight_layout()
plt.show()
print(f"→ Best test accuracy = {max(test_acc):.4f} at k = {best_k}")
print("→ k = 1 overfits (train = 1.0); large k smooths the boundary.")
Section 7.4 — Evaluation Metrics: Beyond Accuracy
Accuracy alone is misleading with imbalanced classes. A model that always predicts "benign" on 63 % benign data gets 63 % accuracy while missing every cancer case.
The Confusion Matrix¶
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | TP | FN |
| Actual Negative | FP | TN |
Key Metrics¶
$$\boxed{\text{Precision} = \frac{TP}{TP + FP}} \qquad \boxed{\text{Recall} = \frac{TP}{TP + FN}}$$
$$\boxed{F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}}$$
$$\boxed{\text{ROC-AUC} = \text{Area under the ROC curve (TPR vs. FPR)}}$$
| Metric | Best When |
|---|---|
| Precision | Cost of FP is high (e.g., spam filter) |
| Recall | Cost of FN is high (e.g., cancer screening) |
| F1 | You need a single balanced metric |
| ROC-AUC | Comparing models across all thresholds |
# === Section 7.4: Confusion matrix & classification report ===
# Use logistic regression on ALL features for evaluation demo
lr_full = LogisticRegression(max_iter=5000, random_state=42)
lr_full.fit(X_bc_train_s, y_bc_train)
y_pred_lr = lr_full.predict(X_bc_test_s)
y_prob_lr = lr_full.predict_proba(X_bc_test_s)[:, 1]
fig, axes = plt.subplots(1, 2, figsize=(12, 4.5))
# 1. Confusion matrix
cm = confusion_matrix(y_bc_test, y_pred_lr)
ConfusionMatrixDisplay(cm, display_labels=["Benign", "Malignant"]).plot(
ax=axes[0], cmap="Blues", colorbar=False)
axes[0].set_title("Confusion Matrix — Logistic Regression")
# 2. Normalised confusion matrix
cm_norm = cm.astype(float) / cm.sum(axis=1, keepdims=True)
ConfusionMatrixDisplay(cm_norm, display_labels=["Benign", "Malignant"]).plot(
ax=axes[1], cmap="Blues", colorbar=False, values_format=".2%")
axes[1].set_title("Normalised Confusion Matrix")
plt.tight_layout()
plt.show()
print(classification_report(y_bc_test, y_pred_lr,
target_names=["Benign", "Malignant"]))
# === Section 7.4 (cont.): ROC curve & Precision–Recall curve ===
fpr, tpr, _ = roc_curve(y_bc_test, y_prob_lr)
auc_val = roc_auc_score(y_bc_test, y_prob_lr)
prec_curve, rec_curve, _ = precision_recall_curve(y_bc_test, y_prob_lr)
ap = average_precision_score(y_bc_test, y_prob_lr)
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
# ROC
axes[0].plot(fpr, tpr, color="steelblue", lw=2,
label=f"Logistic Reg (AUC = {auc_val:.3f})")
axes[0].plot([0, 1], [0, 1], "r--", lw=1, label="Random (AUC = 0.5)")
axes[0].set_xlabel("False Positive Rate")
axes[0].set_ylabel("True Positive Rate")
axes[0].set_title("ROC Curve")
axes[0].legend(loc="lower right")
# PR
axes[1].plot(rec_curve, prec_curve, color="#ef8a62", lw=2,
label=f"Logistic Reg (AP = {ap:.3f})")
axes[1].set_xlabel("Recall")
axes[1].set_ylabel("Precision")
axes[1].set_title("Precision–Recall Curve")
axes[1].legend(loc="lower left")
fig.suptitle("Threshold-Independent Evaluation — Wisconsin BC",
fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
print(f"→ ROC-AUC = {auc_val:.4f} | Average Precision = {ap:.4f}")
print("→ Both curves show the model distinguishes classes very well.")
Section 7.5 — Cross-Validation & Hyperparameter Tuning
Why Cross-Validation?¶
A single train/test split is sensitive to the random split. K-fold cross-validation partitions the data into $k$ folds and trains $k$ models, each time holding out a different fold for validation.
$$\boxed{\text{CV Score} = \frac{1}{k}\sum_{i=1}^{k} \text{Score}_i}$$
Stratified K-Fold¶
For classification, stratified K-fold ensures each fold preserves the class distribution — essential with imbalanced data.
GridSearchCV & RandomizedSearchCV¶
- GridSearchCV: exhaustive search over a parameter grid.
- RandomizedSearchCV: samples a fixed number of random combinations — faster when the grid is large.
Both use cross-validation internally to estimate each configuration's performance.
# === Section 7.5: Stratified K-fold CV comparison ===
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
classifiers = {
"Logistic Regression": LogisticRegression(max_iter=5000, random_state=42),
"Decision Tree (d=4)": DecisionTreeClassifier(max_depth=4, random_state=42),
"KNN (k=5)": KNeighborsClassifier(n_neighbors=5),
"SVM (RBF)": SVC(random_state=42),
}
print("═══ 5-Fold Stratified CV — Wisconsin BC (scaled) ═══\n")
print(f"{'Classifier':25s} {'Mean Acc':>9s} {'Std':>7s} {'Scores'}")
print("─" * 75)
cv_results = {}
for name, clf in classifiers.items():
pipe = Pipeline([("scaler", StandardScaler()), ("clf", clf)])
scores = cross_val_score(pipe, X_bc, y_bc, cv=cv, scoring="accuracy")
cv_results[name] = scores
print(f" {name:25s} {scores.mean():9.4f} {scores.std():7.4f} "
f"{np.round(scores, 4)}")
print("\n→ All classifiers perform well on this dataset (clear class separation).")
# === Section 7.5 (cont.): GridSearchCV for KNN ===
pipe_knn = Pipeline([
("scaler", StandardScaler()),
("knn", KNeighborsClassifier()),
])
param_grid = {
"knn__n_neighbors": [3, 5, 7, 9, 11, 15, 21],
"knn__weights": ["uniform", "distance"],
"knn__metric": ["euclidean", "manhattan"],
}
grid = GridSearchCV(pipe_knn, param_grid, cv=cv, scoring="accuracy",
n_jobs=-1, return_train_score=True)
grid.fit(X_bc, y_bc)
print("═══ GridSearchCV — KNN ═══\n")
print(f" Best params: {grid.best_params_}")
print(f" Best CV accuracy: {grid.best_score_:.4f}")
print(f" Total configs tested: {len(grid.cv_results_['mean_test_score'])}")
# Top 5 configurations
results_df = pd.DataFrame(grid.cv_results_)
top5 = (results_df[["param_knn__n_neighbors", "param_knn__weights",
"param_knn__metric", "mean_test_score", "std_test_score"]]
.sort_values("mean_test_score", ascending=False)
.head(5))
print("\nTop 5 configurations:")
print(top5.to_string(index=False))
Section 7.6 — Model Comparison Pipeline
sklearn.pipeline.Pipeline chains preprocessing and modelling into a
single estimator that is safe to use inside cross-validation (no data
leakage from the scaler).
pipe = Pipeline([
("scaler", StandardScaler()),
("clf", LogisticRegression()),
])
cross_val_score(pipe, X, y, cv=5) # scaler fits only on train folds
We now run a head-to-head comparison of six classifiers using pipelines, plotting their CV score distributions.
# === Section 7.6: Head-to-head model comparison ===
# Strip 'knn__' prefix from GridSearchCV best params for standalone KNN
knn_params = {k.replace("knn__", ""): v for k, v in grid.best_params_.items()}
all_classifiers = {
"Logistic Reg": LogisticRegression(max_iter=5000, random_state=42),
"Decision Tree": DecisionTreeClassifier(max_depth=4, random_state=42),
"KNN (best)": KNeighborsClassifier(**knn_params),
"SVM (RBF)": SVC(random_state=42),
"Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
"Gradient Boosting": GradientBoostingClassifier(n_estimators=100, random_state=42),
}
cv_all = {}
for name, clf in all_classifiers.items():
pipe = Pipeline([("scaler", StandardScaler()), ("clf", clf)])
scores = cross_val_score(pipe, X_bc, y_bc, cv=cv, scoring="accuracy")
cv_all[name] = scores
# Summary table
print("═══ Model Comparison — 5-Fold Stratified CV ═══\n")
print(f"{'Classifier':20s} {'Mean':>7s} {'Std':>6s} {'Min':>6s} {'Max':>6s}")
print("─" * 50)
for name, scores in cv_all.items():
print(f" {name:20s} {scores.mean():7.4f} {scores.std():6.4f} "
f"{scores.min():6.4f} {scores.max():6.4f}")
# === Section 7.6 (cont.): CV score distribution box plot ===
fig, ax = plt.subplots(figsize=(10, 5))
cv_df = pd.DataFrame(cv_all)
cv_df_melted = cv_df.melt(var_name="Classifier", value_name="Accuracy")
sns.boxplot(data=cv_df_melted, x="Classifier", y="Accuracy", ax=ax,
palette=["steelblue", "#67a9cf", "#ef8a62", "#b2df8a",
"#2166ac", "#d1e5f0"])
ax.set_ylim(0.90, 1.005)
ax.set_title("Cross-Validation Accuracy Distribution — Wisconsin BC")
ax.set_xlabel("")
plt.xticks(rotation=20, ha="right")
plt.tight_layout()
plt.show()
best_model = max(cv_all, key=lambda k: cv_all[k].mean())
print(f"→ Best overall: {best_model} "
f"(mean CV accuracy = {cv_all[best_model].mean():.4f})")
Section 7.7 — Case Study: Bank Customer Churn
We now apply the full classification workflow to the Bank Churners dataset — 10,127 credit-card customers with 21 attributes covering demographics, transaction behaviour, and account metrics.
Target: Attrition_Flag (Existing Customer vs. Attrited Customer)
This is a realistic imbalanced dataset (≈ 84 % existing vs. 16 % attrited), which makes it ideal for practising the evaluation techniques from §7.4.
Workflow:
- EDA — distributions, class balance, correlations
- Data Preparation — encoding, scaling, train/test split
- Model Fitting — logistic regression, decision tree, random forest
- Evaluation — confusion matrix, ROC-AUC, PR curve
- Business Recommendations
# === Section 7.7: Load Bank Churn & EDA ===
df_churn = pd.read_csv(DATA / "bankchurn.csv")
print(f"Bank Churn: {df_churn.shape[0]} rows × {df_churn.shape[1]} columns")
print(f"\nTarget distribution:")
print(df_churn["Attrition_Flag"].value_counts())
print(f"\nImbalance ratio: "
f"{df_churn['Attrition_Flag'].value_counts(normalize=True).iloc[0]:.1%} / "
f"{df_churn['Attrition_Flag'].value_counts(normalize=True).iloc[1]:.1%}")
fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))
# 1. Target balance
df_churn["Attrition_Flag"].value_counts().plot(kind="bar", ax=axes[0],
color=["steelblue", "#ef8a62"], edgecolor="white")
axes[0].set_title("Class Distribution")
axes[0].set_ylabel("Count")
axes[0].set_xticklabels(axes[0].get_xticklabels(), rotation=0)
# 2. Age distribution by class
for label, color in zip(["Existing Customer", "Attrited Customer"],
["steelblue", "#ef8a62"]):
axes[1].hist(df_churn.loc[df_churn["Attrition_Flag"] == label, "Customer_Age"],
bins=30, alpha=0.5, color=color, label=label, edgecolor="white")
axes[1].set_xlabel("Customer Age")
axes[1].set_title("Age Distribution by Churn Status")
axes[1].legend(fontsize=8)
# 3. Transaction count vs. total amount
for label, color in zip(["Existing Customer", "Attrited Customer"],
["steelblue", "#ef8a62"]):
mask = df_churn["Attrition_Flag"] == label
axes[2].scatter(df_churn.loc[mask, "Total_Trans_Ct"],
df_churn.loc[mask, "Total_Trans_Amt"],
s=5, alpha=0.3, color=color, label=label)
axes[2].set_xlabel("Total Transaction Count")
axes[2].set_ylabel("Total Transaction Amount")
axes[2].set_title("Transactions by Churn Status")
axes[2].legend(fontsize=8)
fig.suptitle("Bank Customer Churn — Exploratory Analysis", fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
# === Section 7.7: Data preparation ===
df_ch = df_churn.copy()
# Encode target: 1 = Attrited, 0 = Existing
df_ch["Churn"] = (df_ch["Attrition_Flag"] == "Attrited Customer").astype(int)
df_ch.drop(columns=["Attrition_Flag", "CLIENTNUM"], inplace=True)
# One-hot encode categorical columns
cat_cols = df_ch.select_dtypes(include="object").columns.tolist()
df_ch = pd.get_dummies(df_ch, columns=cat_cols, dtype=int)
# Features & target
target_ch = "Churn"
features_ch = [c for c in df_ch.columns if c != target_ch]
X_ch = df_ch[features_ch].values
y_ch = df_ch[target_ch].values
X_ch_train, X_ch_test, y_ch_train, y_ch_test = train_test_split(
X_ch, y_ch, test_size=0.30, random_state=42, stratify=y_ch
)
scaler_ch = StandardScaler().fit(X_ch_train)
X_ch_train_s = scaler_ch.transform(X_ch_train)
X_ch_test_s = scaler_ch.transform(X_ch_test)
print(f"Features: {len(features_ch)}")
print(f"Training set: {X_ch_train_s.shape} | Test set: {X_ch_test_s.shape}")
print(f"Train class balance: {dict(pd.Series(y_ch_train).value_counts())}")
# === Section 7.7: Fit three classifiers ===
models_ch = {
"Logistic Reg": LogisticRegression(max_iter=5000, random_state=42),
"Decision Tree": DecisionTreeClassifier(max_depth=5, random_state=42),
"Random Forest": RandomForestClassifier(n_estimators=200, random_state=42),
}
predictions_ch = {}
probabilities_ch = {}
print("═══ Bank Churn — Model Training ═══\n")
for name, clf in models_ch.items():
clf.fit(X_ch_train_s, y_ch_train)
y_pred = clf.predict(X_ch_test_s)
y_prob = clf.predict_proba(X_ch_test_s)[:, 1]
predictions_ch[name] = y_pred
probabilities_ch[name] = y_prob
acc = accuracy_score(y_ch_test, y_pred)
prec = precision_score(y_ch_test, y_pred)
rec = recall_score(y_ch_test, y_pred)
f1 = f1_score(y_ch_test, y_pred)
auc = roc_auc_score(y_ch_test, y_prob)
print(f" {name:16s} Acc={acc:.4f} Prec={prec:.4f} "
f"Rec={rec:.4f} F1={f1:.4f} AUC={auc:.4f}")
# === Section 7.7: Confusion matrices for all models ===
fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))
for ax, (name, y_pred) in zip(axes, predictions_ch.items()):
cm = confusion_matrix(y_ch_test, y_pred)
ConfusionMatrixDisplay(cm, display_labels=["Existing", "Attrited"]).plot(
ax=ax, cmap="Blues", colorbar=False)
ax.set_title(name)
fig.suptitle("Confusion Matrices — Bank Customer Churn", fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
# === Section 7.7: ROC & PR curves — all models ===
colors = ["steelblue", "#ef8a62", "#b2df8a"]
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
for (name, y_prob), color in zip(probabilities_ch.items(), colors):
# ROC
fpr, tpr, _ = roc_curve(y_ch_test, y_prob)
auc_val = roc_auc_score(y_ch_test, y_prob)
axes[0].plot(fpr, tpr, color=color, lw=2,
label=f"{name} (AUC={auc_val:.3f})")
# PR
prec_c, rec_c, _ = precision_recall_curve(y_ch_test, y_prob)
ap = average_precision_score(y_ch_test, y_prob)
axes[1].plot(rec_c, prec_c, color=color, lw=2,
label=f"{name} (AP={ap:.3f})")
axes[0].plot([0, 1], [0, 1], "r--", lw=1)
axes[0].set_xlabel("False Positive Rate")
axes[0].set_ylabel("True Positive Rate")
axes[0].set_title("ROC Curves")
axes[0].legend(loc="lower right", fontsize=9)
axes[1].set_xlabel("Recall")
axes[1].set_ylabel("Precision")
axes[1].set_title("Precision–Recall Curves")
axes[1].legend(loc="lower left", fontsize=9)
fig.suptitle("Model Comparison — Bank Customer Churn", fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
# === Section 7.7: Feature importance — Random Forest ===
rf_imp = pd.Series(
models_ch["Random Forest"].feature_importances_, index=features_ch
)
top15 = rf_imp.nlargest(15)
fig, ax = plt.subplots(figsize=(9, 6))
top15.sort_values().plot(kind="barh", ax=ax, color="steelblue",
edgecolor="white", linewidth=0.5)
ax.set_xlabel("Feature Importance (Gini)")
ax.set_title("Top 15 Features — Random Forest (Bank Churn)")
plt.tight_layout()
plt.show()
print("→ Transaction count & amount are the strongest churn predictors,")
print(" followed by revolving balance and relationship count.")
# === Section 7.7: Key findings & business recommendations ===
print("═══ Key Findings — Bank Customer Churn ═══\n")
best_name = max(probabilities_ch,
key=lambda k: roc_auc_score(y_ch_test, probabilities_ch[k]))
best_auc = roc_auc_score(y_ch_test, probabilities_ch[best_name])
best_f1 = f1_score(y_ch_test, predictions_ch[best_name])
print(f" Best model: {best_name}")
print(f" ROC-AUC = {best_auc:.4f}")
print(f" F1-score = {best_f1:.4f}")
print(f"\n Top 5 churn predictors:")
for feat, imp in rf_imp.nlargest(5).items():
print(f" → {feat}: importance = {imp:.4f}")
print("\n═══ Business Recommendations ═══")
print("→ Transaction frequency and total amount are the strongest")
print(" signals of churn — declining engagement predicts attrition.")
print("→ Target customers with dropping transaction counts for")
print(" proactive retention campaigns (rewards, personal outreach).")
print("→ Monitor revolving balance and credit utilisation as")
print(" secondary indicators of disengagement.")
print("→ The Random Forest / Gradient Boosting models provide the")
print(" best discrimination — deploy with a recall-optimised threshold")
print(" to catch more at-risk customers (accept slightly more FPs).")
Section 7.8 — Summary & Quick Reference
Key Formulas¶
| Quantity | Formula |
|---|---|
| Sigmoid | $\sigma(z) = \dfrac{1}{1 + e^{-z}}$ |
| Log-odds | $\log\dfrac{p}{1-p} = \mathbf{x}^T \boldsymbol{\beta}$ |
| Gini impurity | $G = 1 - \sum p_k^2$ |
| Euclidean distance | $d = \sqrt{\sum(a_j - b_j)^2}$ |
| Precision | $TP / (TP + FP)$ |
| Recall | $TP / (TP + FN)$ |
| $F_1$ score | $2 \cdot \dfrac{P \cdot R}{P + R}$ |
| K-fold CV | $\dfrac{1}{k}\sum \text{Score}_i$ |
Classifier Decision Guide¶
| Situation | Recommended Classifier |
|---|---|
| Baseline / interpretable coefficients | Logistic Regression |
| Non-linear boundaries, few features | Decision Tree / KNN |
| High-dimensional, clear margin | SVM (RBF kernel) |
| Best accuracy, many features | Random Forest / Gradient Boosting |
| Need probability calibration | Logistic Regression or calibrated ensemble |
Python Cheat Sheet¶
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, roc_auc_score
# Pipeline: pipe = Pipeline([("scaler", StandardScaler()), ("clf", model)])
# CV: cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
# Grid: GridSearchCV(pipe, param_grid, cv=5).fit(X, y)
# Report: classification_report(y_true, y_pred)
# ROC-AUC: roc_auc_score(y_true, y_prob)
Key Takeaways¶
$$\boxed{\text{Accuracy is not enough — use precision, recall, F1, and ROC-AUC for classification.}}$$
- Logistic regression is the classification analogue of linear regression — start here for interpretability and baseline performance.
- Decision trees are intuitive but overfit easily; random forests and gradient boosting fix this with ensembling.
- KNN is simple but scaling-sensitive and slow on large datasets.
- Cross-validation and pipelines make your workflow rigorous and prevent data leakage.
- The case study demonstrated that on imbalanced data, the model with the highest accuracy is not necessarily the best — ROC-AUC and F1 give a more honest picture.
Chapter 6 predicted how much; this chapter predicts which class. Together they cover the two pillars of supervised learning.
Next → Chapter 8: Unsupervised Learning
References¶
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.). Springer. Chapter 4 (classification) and Chapter 5 (resampling/model selection). https://www.statlearning.com/
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. Chapter 4 (classification) and Chapter 7 (model assessment and selection).
Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of IJCAI, 14(2), 1137-1145. The classical reference for k-fold cross-validation.
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273-297. The original SVM paper.
scikit-learn Developers. scikit-learn
model_selectionand metrics documentation (GridSearchCV, ROC/AUC, confusion matrix). https://scikit-learn.org/stable/modules/model_selection.html