Statistical Basis of AI/ML — Unsupervised Learning
Chapter 8 — Discovering Hidden Structure in Data
Why This Chapter?
In Chapters 6 and 7 you built models where every observation had a known label — price, diagnosis, churn. But most real-world data arrives unlabelled. No one tells you how many customer segments exist, which genes co-vary, or which sensor readings form natural groups.
Unsupervised learning lets the data speak for itself. Algorithms find patterns — clusters, latent dimensions, outliers — without any target variable. The results drive customer segmentation, anomaly detection, feature compression, and exploratory analysis across every domain from oncology to e-commerce.
This chapter covers the three pillars: clustering (K-Means, hierarchical, DBSCAN), dimensionality reduction (PCA), and the evaluation metrics needed when you have no ground truth.
What You Will Learn
| $\S$ | Topic | Description |
|---|---|---|
| 8.0 | Setup | Imports, GPU check, plot configuration |
| 8.1 | What Is Unsupervised Learning? | Supervised vs. unsupervised, taxonomy, when to use it |
| 8.2 | K-Means Clustering | Algorithm, elbow method, silhouette analysis |
| 8.3 | Hierarchical Clustering | Agglomerative method, linkage criteria, dendrograms |
| 8.4 | DBSCAN | Density-based clustering, $\varepsilon$ and min_samples, noise detection |
| 8.5 | Principal Component Analysis | Eigenvalues, explained variance, biplots |
| 8.6 | Cluster Evaluation Metrics | Silhouette, Calinski–Harabasz, Davies–Bouldin |
| 8.7 | Case Study — Wine Quality Segmentation | Scale → PCA → K-Means → interpretation |
| 8.8 | Summary & Quick Reference | Formulas, decision guide, cheat sheet |
Section 8.0 — Setup: Imports & Configuration
This chapter adds clustering and decomposition tools from scikit-learn.
| Library | Role in This Chapter |
|---|---|
| NumPy | Array arithmetic, random seeding |
| Pandas | DataFrames, data wrangling |
| Matplotlib | Cluster scatter plots, dendrograms, scree plots |
| Seaborn | Heat maps, pair plots |
| scikit-learn | K-Means, Agglomerative, DBSCAN, PCA, metrics |
| SciPy | Linkage & dendrogram for hierarchical clustering |
| PyTorch | GPU availability check |
# === Chapter 8: Setup ===
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import scipy
from pathlib import Path
from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import (
silhouette_score, silhouette_samples,
calinski_harabasz_score, davies_bouldin_score,
)
from sklearn.datasets import make_blobs, make_moons
import sklearn
from scipy.cluster.hierarchy import dendrogram, linkage
# ── Plot defaults (house style) ──
plt.rcParams.update({
"figure.figsize": (8, 4),
"axes.grid": True,
"grid.alpha": 0.3,
"axes.spines.top": False,
"axes.spines.right": False,
})
sns.set_theme(style="whitegrid")
rng = np.random.default_rng(42)
DATA = Path("../../data")
print(f"NumPy {np.__version__} | Pandas {pd.__version__} | "
f"SciPy {scipy.__version__}")
print(f"scikit-learn {sklearn.__version__}")
print("\nChapter 8 environment ready.")
Section 8.1 — What Is Unsupervised Learning?
Supervised vs. Unsupervised¶
| Aspect | Supervised | Unsupervised |
|---|---|---|
| Input | Features $X$ and labels $y$ | Features $X$ only |
| Goal | Predict $y$ from $X$ | Discover structure in $X$ |
| Evaluation | Compare predictions to known labels | Internal metrics (silhouette, variance explained) |
| Examples | Regression, classification | Clustering, PCA, anomaly detection |
Taxonomy of Unsupervised Methods¶
$$\boxed{\text{Unsupervised Learning} \begin{cases} \text{Clustering} & \text{group similar observations} \\ \text{Dimensionality Reduction} & \text{compress features, retain variance} \\ \text{Anomaly Detection} & \text{identify outliers} \end{cases}}$$
When to Use Unsupervised Learning¶
- No labels available — most raw data lacks human annotation.
- Exploratory analysis — understand natural groupings before building a supervised model.
- Feature engineering — PCA components or cluster labels become inputs to downstream models.
- Compression — reduce 30 features to 3 principal components for visualisation.
We begin with the most widely used method: K-Means clustering.
Section 8.2 — K-Means Clustering
The Algorithm¶
K-Means partitions $n$ observations into $K$ clusters by minimising within-cluster sum of squares (WCSS, also called inertia):
$$\boxed{\text{WCSS} = \sum_{k=1}^{K} \sum_{\mathbf{x}_i \in C_k} \|\mathbf{x}_i - \boldsymbol{\mu}_k\|^2}$$
Steps (Lloyd's algorithm):
- Randomly initialise $K$ centroids.
- Assign each point to the nearest centroid.
- Update each centroid to the mean of its assigned points.
- Repeat steps 2–3 until convergence.
Choosing $K$: The Elbow Method
Plot WCSS against $K$. The "elbow" — where adding another cluster yields diminishing returns — suggests a good $K$.
Silhouette Analysis¶
For each point $i$:
$$\boxed{s(i) = \frac{b(i) - a(i)}{\max\{a(i),\, b(i)\}}}$$
- $a(i)$ = mean distance to points in the same cluster.
- $b(i)$ = mean distance to points in the nearest other cluster.
- $s(i) \in [-1, 1]$; higher is better.
We demonstrate on the Wine Quality dataset (4 898 wines, 11 physicochemical features).
# === Section 8.2: Load Wine Quality & K-Means clustering ===
df_wine = pd.read_csv(DATA / "wine_data.csv")
print(f"Wine Quality dataset: {df_wine.shape[0]} rows × {df_wine.shape[1]} columns")
print(f"Columns: {df_wine.columns.tolist()}")
# Separate features (drop the quality rating — we won't use it as a label)
X_wine = df_wine.drop(columns=["quality"])
# Scale features — essential for distance-based methods
scaler_wine = StandardScaler()
X_wine_sc = scaler_wine.fit_transform(X_wine)
print(f"Feature matrix after scaling: {X_wine_sc.shape}")
print("→ Quality column excluded; all 11 physicochemical features retained.")
# === Section 8.2 (cont.): Elbow method ===
K_range = range(2, 11)
inertias = []
for k in K_range:
km = KMeans(n_clusters=k, n_init=10, random_state=42)
km.fit(X_wine_sc)
inertias.append(km.inertia_)
fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(list(K_range), inertias, "o-", color="steelblue", linewidth=2)
ax.set_xlabel("Number of clusters (K)")
ax.set_ylabel("WCSS (inertia)")
ax.set_title("Elbow Method — Wine Quality", fontsize=13)
ax.set_xticks(list(K_range))
plt.tight_layout()
plt.show()
print("→ The elbow appears around K = 3–4, suggesting 3 or 4 natural groupings.")
# === Section 8.2 (cont.): Silhouette analysis for K = 2..6 ===
sil_scores = []
for k in range(2, 7):
km = KMeans(n_clusters=k, n_init=10, random_state=42)
labels = km.fit_predict(X_wine_sc)
sc = silhouette_score(X_wine_sc, labels)
sil_scores.append((k, sc))
print(f" K = {k} → Silhouette = {sc:.4f}")
best_k = max(sil_scores, key=lambda x: x[1])
print(f"\n→ Best silhouette at K = {best_k[0]} ({best_k[1]:.4f}).")
# === Section 8.2 (cont.): K-Means scatter (first 2 PCA components) ===
km_final = KMeans(n_clusters=3, n_init=10, random_state=42)
wine_labels = km_final.fit_predict(X_wine_sc)
# Project to 2-D for visualisation
pca_2d = PCA(n_components=2, random_state=42)
X_2d = pca_2d.fit_transform(X_wine_sc)
palette = ["steelblue", "#ef8a62", "#b2df8a"]
fig, ax = plt.subplots(figsize=(8, 5))
for cl in range(3):
mask = wine_labels == cl
ax.scatter(X_2d[mask, 0], X_2d[mask, 1],
c=palette[cl], label=f"Cluster {cl}",
alpha=0.5, edgecolor="white", s=30)
ax.set_xlabel(f"PC 1 ({pca_2d.explained_variance_ratio_[0]:.1%} var)")
ax.set_ylabel(f"PC 2 ({pca_2d.explained_variance_ratio_[1]:.1%} var)")
ax.set_title("K-Means (K = 3) — Wine Quality (PCA projection)", fontsize=13)
ax.legend()
plt.tight_layout()
plt.show()
print(f"→ 3 clusters visualised on the first two principal components "
f"({pca_2d.explained_variance_ratio_[:2].sum():.1%} total variance).")
Section 8.3 — Hierarchical Clustering
Agglomerative (Bottom-Up) Algorithm¶
- Start: every observation is its own cluster.
- Merge the two closest clusters.
- Repeat until one cluster remains.
Linkage Criteria¶
| Linkage | Distance Between Clusters |
|---|---|
| Single | Minimum pairwise distance |
| Complete | Maximum pairwise distance |
| Average | Mean pairwise distance |
| Ward | Increase in WCSS after merge |
Ward linkage minimises variance (like K-Means) and is the default choice for compact, spherical clusters.
Dendrograms¶
A dendrogram visualises the merge history. Cutting at a chosen height produces $K$ clusters. Unlike K-Means, hierarchical clustering does not require you to specify $K$ in advance — you read it from the dendrogram.
We subsample the wine dataset for a readable dendrogram, then fit agglomerative clustering on the full data.
# === Section 8.3: Dendrogram (subsample for readability) ===
idx_sub = rng.choice(len(X_wine_sc), size=200, replace=False)
X_sub = X_wine_sc[idx_sub]
Z = linkage(X_sub, method="ward")
fig, ax = plt.subplots(figsize=(12, 5))
dendrogram(Z, truncate_mode="lastp", p=30,
leaf_rotation=90, leaf_font_size=9,
ax=ax, color_threshold=15)
ax.set_title("Ward Dendrogram — Wine Quality (200-sample subset)", fontsize=13)
ax.set_xlabel("Cluster (or sample count in parentheses)")
ax.set_ylabel("Ward distance")
ax.axhline(y=15, color="#ef8a62", linestyle="--", label="Cut at distance 15")
ax.legend()
plt.tight_layout()
plt.show()
print("→ Cutting the dendrogram at distance ≈ 15 yields ~3 clusters, "
"consistent with the K-Means elbow analysis.")
# === Section 8.3 (cont.): Agglomerative clustering on full data ===
agg = AgglomerativeClustering(n_clusters=3, linkage="ward")
agg_labels = agg.fit_predict(X_wine_sc)
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
titles = ["K-Means (K = 3)", "Agglomerative Ward (K = 3)"]
label_sets = [wine_labels, agg_labels]
for ax, title, labs in zip(axes, titles, label_sets):
for cl in range(3):
mask = labs == cl
ax.scatter(X_2d[mask, 0], X_2d[mask, 1],
c=palette[cl], label=f"Cluster {cl}",
alpha=0.5, edgecolor="white", s=30)
ax.set_xlabel("PC 1")
ax.set_ylabel("PC 2")
ax.set_title(title, fontsize=12)
ax.legend(fontsize=9)
fig.suptitle("K-Means vs. Hierarchical Clustering — Wine Quality",
fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
sil_km = silhouette_score(X_wine_sc, wine_labels)
sil_agg = silhouette_score(X_wine_sc, agg_labels)
print(f"Silhouette K-Means = {sil_km:.4f} | Agglomerative = {sil_agg:.4f}")
print("→ Both methods produce comparable cluster assignments on this dataset.")
Section 8.4 — DBSCAN
Density-Based Spatial Clustering¶
K-Means and hierarchical methods assume convex, roughly spherical clusters. Real data often contains irregular shapes and noise. DBSCAN (Density-Based Spatial Clustering of Applications with Noise) finds clusters of arbitrary shape by linking dense regions.
Key Parameters¶
| Parameter | Meaning |
|---|---|
$\varepsilon$ (eps) |
Maximum distance between two neighbours |
min_samples |
Minimum points to form a dense core |
Point Categories¶
- Core point: has $\ge$
min_samplesneighbours within $\varepsilon$. - Border point: within $\varepsilon$ of a core point but not itself core.
- Noise point: neither core nor border — labelled $-1$.
Strengths & Weaknesses¶
| Strength | Weakness |
|---|---|
| Finds arbitrary-shape clusters | Sensitive to $\varepsilon$ and min_samples |
| Identifies noise/outliers | Struggles with varying density |
| No need to specify $K$ | $O(n^2)$ without spatial index |
We demonstrate on synthetic data where K-Means fails — half-moon shapes — then apply DBSCAN to the wine data.
# === Section 8.4: DBSCAN on synthetic half-moons ===
X_moons, y_moons = make_moons(n_samples=500, noise=0.08, random_state=42)
km_moons = KMeans(n_clusters=2, n_init=10, random_state=42)
km_moon_labels = km_moons.fit_predict(X_moons)
db_moons = DBSCAN(eps=0.2, min_samples=5)
db_moon_labels = db_moons.fit_predict(X_moons)
fig, axes = plt.subplots(1, 3, figsize=(16, 4))
# Ground truth
for c in [0, 1]:
mask = y_moons == c
axes[0].scatter(X_moons[mask, 0], X_moons[mask, 1],
c=palette[c], alpha=0.6, edgecolor="white", s=30)
axes[0].set_title("Ground Truth", fontsize=12)
# K-Means
for c in [0, 1]:
mask = km_moon_labels == c
axes[1].scatter(X_moons[mask, 0], X_moons[mask, 1],
c=palette[c], alpha=0.6, edgecolor="white", s=30)
axes[1].set_title("K-Means (K = 2)", fontsize=12)
# DBSCAN
unique_db = sorted(set(db_moon_labels))
db_palette = {-1: "gray"}
ci = 0
for lab in unique_db:
if lab == -1:
continue
db_palette[lab] = palette[ci % len(palette)]
ci += 1
for lab in unique_db:
mask = db_moon_labels == lab
lbl = "Noise" if lab == -1 else f"Cluster {lab}"
axes[2].scatter(X_moons[mask, 0], X_moons[mask, 1],
c=db_palette[lab], alpha=0.6,
edgecolor="white", s=30, label=lbl)
axes[2].set_title("DBSCAN (ε = 0.2)", fontsize=12)
axes[2].legend(fontsize=9)
fig.suptitle("K-Means vs. DBSCAN on Non-Convex Data",
fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
n_noise = (db_moon_labels == -1).sum()
print(f"DBSCAN found {len(set(db_moon_labels) - {-1})} clusters "
f"and {n_noise} noise points.")
print("→ DBSCAN correctly separates the moons; K-Means cannot.")
# === Section 8.4 (cont.): DBSCAN on wine data ===
db_wine = DBSCAN(eps=3.5, min_samples=10)
db_wine_labels = db_wine.fit_predict(X_wine_sc)
n_clusters_db = len(set(db_wine_labels) - {-1})
n_noise_db = (db_wine_labels == -1).sum()
fig, ax = plt.subplots(figsize=(8, 5))
unique_labels = sorted(set(db_wine_labels))
extended_palette = ["gray"] + palette + ["#67a9cf", "#2166ac", "#d1e5f0"]
for i, lab in enumerate(unique_labels):
mask = db_wine_labels == lab
lbl = "Noise" if lab == -1 else f"Cluster {lab}"
ax.scatter(X_2d[mask, 0], X_2d[mask, 1],
c=extended_palette[i % len(extended_palette)],
alpha=0.5, edgecolor="white", s=30, label=lbl)
ax.set_xlabel("PC 1")
ax.set_ylabel("PC 2")
ax.set_title(f"DBSCAN on Wine Quality — {n_clusters_db} clusters, "
f"{n_noise_db} noise points", fontsize=13)
ax.legend(fontsize=9)
plt.tight_layout()
plt.show()
print(f"DBSCAN (eps=3.5, min_samples=10): {n_clusters_db} clusters, "
f"{n_noise_db} noise points ({n_noise_db/len(db_wine_labels):.1%} of data).")
print("→ DBSCAN is sensitive to epsilon; tuning is critical for high-dimensional data.")
Section 8.5 — Principal Component Analysis (PCA)
The Idea¶
PCA finds new axes (principal components) that capture the maximum variance in the data. The first component captures the most variance, the second captures the most remaining variance (orthogonal to the first), and so on.
The Math¶
Given a centred data matrix $\mathbf{X}$ ($n \times p$), PCA computes the eigendecomposition of the covariance matrix:
$$\boxed{\mathbf{C} = \frac{1}{n-1}\mathbf{X}^\top \mathbf{X} = \mathbf{V}\,\boldsymbol{\Lambda}\,\mathbf{V}^\top}$$
- $\mathbf{V}$: matrix of eigenvectors (the principal-component directions).
- $\boldsymbol{\Lambda}$: diagonal matrix of eigenvalues (variance along each component).
Explained Variance Ratio¶
$$\boxed{\text{EVR}_j = \frac{\lambda_j}{\sum_{k=1}^{p} \lambda_k}}$$
A scree plot of cumulative EVR shows how many components are needed to retain a target fraction (e.g., 95 %) of total variance.
We apply PCA to the Wisconsin Breast Cancer dataset (30 numeric features) — the same dataset from Chapter 7, now used without labels.
# === Section 8.5: PCA on Wisconsin Breast Cancer ===
df_bc = pd.read_csv(DATA / "wisc_bc_data.csv")
print(f"Wisconsin BC dataset: {df_bc.shape[0]} rows × {df_bc.shape[1]} columns")
# Use only numeric features (drop diagnosis)
X_bc = df_bc.drop(columns=["diagnosis"])
y_bc = df_bc["diagnosis"] # kept for colouring only — NOT used in PCA
scaler_bc = StandardScaler()
X_bc_sc = scaler_bc.fit_transform(X_bc)
# Full PCA — keep all 30 components
pca_full = PCA(random_state=42)
X_bc_pca = pca_full.fit_transform(X_bc_sc)
print(f"\nExplained variance by first 5 components:")
for i in range(5):
print(f" PC {i+1}: {pca_full.explained_variance_ratio_[i]:.4f}")
cum_95 = np.argmax(np.cumsum(pca_full.explained_variance_ratio_) >= 0.95) + 1
print(f"\n→ {cum_95} components needed to reach 95% explained variance "
f"(out of {X_bc.shape[1]} original features).")
# === Section 8.5 (cont.): Scree plot & cumulative variance ===
evr = pca_full.explained_variance_ratio_
cum_evr = np.cumsum(evr)
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Scree plot
axes[0].bar(range(1, len(evr) + 1), evr, color="steelblue",
edgecolor="white")
axes[0].set_xlabel("Principal Component")
axes[0].set_ylabel("Explained Variance Ratio")
axes[0].set_title("Scree Plot", fontsize=12)
# Cumulative
axes[1].plot(range(1, len(cum_evr) + 1), cum_evr, "o-",
color="steelblue", linewidth=2)
axes[1].axhline(y=0.95, color="#ef8a62", linestyle="--",
label="95% threshold")
axes[1].axvline(x=cum_95, color="#ef8a62", linestyle=":", alpha=0.5)
axes[1].set_xlabel("Number of Components")
axes[1].set_ylabel("Cumulative Explained Variance")
axes[1].set_title("Cumulative Variance Explained", fontsize=12)
axes[1].legend()
fig.suptitle("PCA — Wisconsin Breast Cancer (30 features)",
fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
print(f"→ The first 2 components alone explain "
f"{cum_evr[1]:.1%} of the total variance.")
# === Section 8.5 (cont.): 2-D PCA projection coloured by diagnosis ===
fig, ax = plt.subplots(figsize=(8, 5))
for label, colour, name in [("M", "#ef8a62", "Malignant"),
("B", "steelblue", "Benign")]:
mask = y_bc == label
ax.scatter(X_bc_pca[mask, 0], X_bc_pca[mask, 1],
c=colour, label=name, alpha=0.5,
edgecolor="white", s=30)
ax.set_xlabel(f"PC 1 ({evr[0]:.1%} variance)")
ax.set_ylabel(f"PC 2 ({evr[1]:.1%} variance)")
ax.set_title("PCA Projection — Wisconsin BC", fontsize=13)
ax.legend()
plt.tight_layout()
plt.show()
print("→ Even without using the diagnosis label, PCA separates the two "
"classes along PC 1 — evidence that the 30 features contain a "
"strong underlying signal.")
# === Section 8.5 (cont.): PCA loadings — top feature contributions ===
loadings = pd.DataFrame(
pca_full.components_[:2].T,
columns=["PC 1", "PC 2"],
index=X_bc.columns,
)
loadings["abs_PC1"] = loadings["PC 1"].abs()
top_features = loadings.nlargest(10, "abs_PC1")[["PC 1", "PC 2"]]
fig, ax = plt.subplots(figsize=(8, 5))
top_features["PC 1"].plot.barh(ax=ax, color="steelblue", edgecolor="white")
ax.set_xlabel("Loading on PC 1")
ax.set_title("Top 10 Feature Loadings on PC 1", fontsize=13)
ax.invert_yaxis()
plt.tight_layout()
plt.show()
print("→ Features related to size (worst area, mean area, worst perimeter) "
"dominate PC 1, confirming that tumour size is the primary axis "
"of variation.")
Section 8.6 — Cluster Evaluation Metrics
Without ground-truth labels, how do we know a clustering is good? Three widely used internal metrics:
Silhouette Score¶
$$\boxed{S = \frac{1}{n}\sum_{i=1}^{n} s(i), \qquad s(i) = \frac{b(i)-a(i)}{\max\{a(i),b(i)\}}}$$
| Value | Interpretation |
|---|---|
| $S \approx 1$ | Dense, well-separated clusters |
| $S \approx 0$ | Overlapping clusters |
| $S < 0$ | Points likely assigned to wrong cluster |
Calinski–Harabasz Index (Variance Ratio)¶
$$\boxed{\text{CH} = \frac{\text{tr}(\mathbf{B}_K)}{\text{tr}(\mathbf{W}_K)} \cdot \frac{n - K}{K - 1}}$$
Higher is better. $\mathbf{B}_K$ = between-cluster dispersion, $\mathbf{W}_K$ = within-cluster dispersion.
Davies–Bouldin Index¶
$$\boxed{\text{DB} = \frac{1}{K}\sum_{i=1}^{K} \max_{j \ne i} \frac{\sigma_i + \sigma_j}{d(c_i, c_j)}}$$
Lower is better. $\sigma_i$ = average distance within cluster $i$, $d(c_i, c_j)$ = distance between centroids.
# === Section 8.6: Compare metrics across K for K-Means on wine data ===
results = []
for k in range(2, 8):
km = KMeans(n_clusters=k, n_init=10, random_state=42)
labs = km.fit_predict(X_wine_sc)
results.append({
"K": k,
"Silhouette": silhouette_score(X_wine_sc, labs),
"Calinski-Harabasz": calinski_harabasz_score(X_wine_sc, labs),
"Davies-Bouldin": davies_bouldin_score(X_wine_sc, labs),
})
df_metrics = pd.DataFrame(results).set_index("K")
print("═══ Cluster Evaluation Metrics (K-Means, Wine Quality) ═══\n")
print(df_metrics.to_string(float_format="{:.4f}".format))
print("─" * 55)
print(f"Best Silhouette: K = {df_metrics['Silhouette'].idxmax()}")
print(f"Best Calinski-Harabasz: K = {df_metrics['Calinski-Harabasz'].idxmax()}")
print(f"Best Davies-Bouldin: K = {df_metrics['Davies-Bouldin'].idxmin()}")
# === Section 8.6 (cont.): Metrics visualisation ===
fig, axes = plt.subplots(1, 3, figsize=(16, 4))
axes[0].plot(df_metrics.index, df_metrics["Silhouette"],
"o-", color="steelblue", linewidth=2)
axes[0].set_title("Silhouette Score (↑)", fontsize=12)
axes[0].set_xlabel("K")
axes[1].plot(df_metrics.index, df_metrics["Calinski-Harabasz"],
"o-", color="#ef8a62", linewidth=2)
axes[1].set_title("Calinski–Harabasz Index (↑)", fontsize=12)
axes[1].set_xlabel("K")
axes[2].plot(df_metrics.index, df_metrics["Davies-Bouldin"],
"o-", color="#b2df8a", linewidth=2)
axes[2].set_title("Davies–Bouldin Index (↓)", fontsize=12)
axes[2].set_xlabel("K")
fig.suptitle("Cluster Evaluation Metrics — Wine Quality",
fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
print("→ All three metrics converge on K = 2 as the statistically "
"cleanest partition; K = 3 offers a pragmatic middle ground.")
Section 8.7 — Case Study: Wine Quality Segmentation
Objective¶
Combine the tools from this chapter into a complete unsupervised pipeline:
- Scale features.
- Reduce dimensions with PCA.
- Cluster with K-Means.
- Profile each cluster by comparing feature means.
- Interpret — what do the segments mean for a winemaker?
This mirrors a real-world workflow: compress noisy, high-dimensional data, discover segments, then translate findings into actionable insight.
# === Section 8.7: EDA — Wine Quality distributions ===
fig, axes = plt.subplots(2, 3, figsize=(16, 8))
plot_cols = ["alcohol", "volatile acidity", "residual sugar",
"pH", "sulphates", "density"]
for ax, col in zip(axes.ravel(), plot_cols):
ax.hist(df_wine[col], bins=40, color="steelblue",
edgecolor="white", alpha=0.8)
ax.set_title(col, fontsize=11)
ax.set_ylabel("Count")
fig.suptitle("Feature Distributions — Wine Quality",
fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
print("→ Residual sugar is heavily right-skewed; density is near-normal.")
print(" Scaling is essential to prevent high-variance features from "
"dominating distance-based algorithms.")
# === Section 8.7 (cont.): PCA → K-Means pipeline ===
# Step 1-2: Scale and PCA (retain 95% variance)
pipe = Pipeline([
("scaler", StandardScaler()),
("pca", PCA(n_components=0.95, random_state=42)),
])
X_wine_pca = pipe.fit_transform(X_wine)
n_comp = pipe.named_steps["pca"].n_components_
print(f"PCA retained {n_comp} components (95% variance threshold).")
# Step 3: K-Means (K = 3 based on earlier analysis)
km_pipe = KMeans(n_clusters=3, n_init=10, random_state=42)
seg_labels = km_pipe.fit_predict(X_wine_pca)
df_wine["Segment"] = seg_labels
print(f"\nSegment sizes:")
print(df_wine["Segment"].value_counts().sort_index().to_string())
print(f"\nSilhouette score: {silhouette_score(X_wine_pca, seg_labels):.4f}")
# === Section 8.7 (cont.): Cluster profiles — feature means by segment ===
profile = df_wine.groupby("Segment").mean(numeric_only=True)
print("═══ Segment Profiles — Mean Feature Values ═══\n")
print(profile.round(3).to_string())
print("─" * 80)
# === Section 8.7 (cont.): Radar-style comparison (bar chart) ===
# Standardise profiles for comparable bar heights
profile_sc = (profile - profile.mean()) / profile.std()
feature_subset = ["alcohol", "volatile acidity", "residual sugar",
"pH", "sulphates", "density"]
fig, ax = plt.subplots(figsize=(12, 5))
x = np.arange(len(feature_subset))
width = 0.25
for seg in range(3):
vals = profile_sc.loc[seg, feature_subset].values
ax.bar(x + seg * width, vals, width,
color=palette[seg], edgecolor="white",
label=f"Segment {seg}")
ax.set_xticks(x + width)
ax.set_xticklabels(feature_subset, rotation=25, ha="right")
ax.set_ylabel("Standardised Mean")
ax.set_title("Segment Profiles — Key Features", fontsize=13)
ax.legend()
ax.axhline(0, color="gray", linewidth=0.8)
plt.tight_layout()
plt.show()
# === Section 8.7 (cont.): PCA scatter by segment ===
fig, ax = plt.subplots(figsize=(8, 5))
for seg in range(3):
mask = seg_labels == seg
ax.scatter(X_wine_pca[mask, 0], X_wine_pca[mask, 1],
c=palette[seg], label=f"Segment {seg}",
alpha=0.4, edgecolor="white", s=30)
evr_pipe = pipe.named_steps["pca"].explained_variance_ratio_
ax.set_xlabel(f"PC 1 ({evr_pipe[0]:.1%} variance)")
ax.set_ylabel(f"PC 2 ({evr_pipe[1]:.1%} variance)")
ax.set_title("Wine Quality Segments (PCA + K-Means)", fontsize=13)
ax.legend()
plt.tight_layout()
plt.show()
# === Section 8.7 (cont.): Quality distribution by segment ===
fig, axes = plt.subplots(1, 3, figsize=(14, 4))
for seg in range(3):
subset = df_wine[df_wine["Segment"] == seg]["quality"]
axes[seg].hist(subset, bins=range(3, 10), color=palette[seg],
edgecolor="white", alpha=0.8)
axes[seg].set_title(f"Segment {seg} (n = {len(subset)})", fontsize=12)
axes[seg].set_xlabel("Quality Score")
axes[seg].set_ylabel("Count")
fig.suptitle("Quality Distribution by Unsupervised Segment",
fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
print("═══ Mean Quality by Segment ═══")
for seg in range(3):
mq = df_wine[df_wine["Segment"] == seg]["quality"].mean()
print(f" Segment {seg}: mean quality = {mq:.2f}")
# === Section 8.7 (cont.): Key findings & recommendations ===
print("═══ Wine Segmentation — Key Findings ═══\n")
print("1. Three segments emerge from physicochemical features alone,")
print(" without using the quality rating as input.\n")
print("2. PCA compressed 11 features into a compact representation")
print(f" ({n_comp} components, 95% variance retained).\n")
print("3. Segment profiles reveal distinct chemical fingerprints:")
print(" • Differences in alcohol, residual sugar, and volatile acidity")
print(" drive the separation.\n")
print("4. The unsupervised segments partially correlate with quality,")
print(" suggesting that chemical composition is predictive of")
print(" perceived quality — a hypothesis testable in Chapter 7's")
print(" classification framework.\n")
print("5. Business recommendation: use segment labels as a new feature")
print(" in supervised models, or as a basis for targeted quality")
print(" improvement programmes.")
# Clean up the temporary column
df_wine.drop(columns=["Segment"], inplace=True)
Section 8.8 — Summary & Quick Reference
Key Formulas¶
| Concept | Formula |
|---|---|
| K-Means WCSS | $\displaystyle \sum_{k=1}^{K}\sum_{\mathbf{x}_i \in C_k}\|\mathbf{x}_i - \boldsymbol{\mu}_k\|^2$ |
| Silhouette | $\displaystyle s(i) = \frac{b(i) - a(i)}{\max\{a(i),b(i)\}}$ |
| PCA covariance | $\displaystyle \mathbf{C} = \frac{1}{n-1}\mathbf{X}^\top\mathbf{X}$ |
| Explained variance | $\displaystyle \text{EVR}_j = \frac{\lambda_j}{\sum_k \lambda_k}$ |
| Calinski–Harabasz | $\displaystyle \frac{\text{tr}(B_K)}{\text{tr}(W_K)}\cdot\frac{n-K}{K-1}$ |
| Davies–Bouldin | $\displaystyle \frac{1}{K}\sum_i \max_{j\ne i}\frac{\sigma_i+\sigma_j}{d(c_i,c_j)}$ |
Algorithm Decision Guide¶
| Algorithm | Best When | Watch Out For |
|---|---|---|
| K-Means | Spherical clusters, known $K$ | Sensitive to scale and outliers |
| Hierarchical | Want a dendrogram, unknown $K$ | $O(n^2)$ memory; impractical for large $n$ |
| DBSCAN | Arbitrary shapes, noise present | Sensitive to $\varepsilon$; struggles with varying density |
| PCA | Feature compression, visualisation | Linear only; loses non-linear structure |
Python Cheat Sheet¶
from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score
# Scale → PCA → K-Means pipeline
pipe = Pipeline([("scaler", StandardScaler()),
("pca", PCA(n_components=0.95)),
("kmeans", KMeans(n_clusters=3, n_init=10))])
labels = pipe.fit_predict(X)
print(silhouette_score(pipe[:-1].transform(X), labels))
Key Takeaways¶
$$\boxed{\text{Unsupervised learning discovers structure without labels — scale your data, choose the right algorithm, and validate with internal metrics.}}$$
- Always scale before distance-based methods (K-Means, hierarchical, DBSCAN).
- Use the elbow method + silhouette to choose $K$; confirm with Calinski–Harabasz and Davies–Bouldin.
- PCA compresses features while preserving variance — inspect the scree plot and loadings.
- DBSCAN excels on non-convex data and naturally flags outliers.
- Cluster labels make powerful engineered features for downstream supervised models.
Chapter 7 gave you supervised classification. Chapter 8 showed you how to find structure when labels don't exist.
Next → Chapter 9: Ensembles & Tree Methods
References¶
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. Chapter 14 covers clustering and PCA in depth.
Jolliffe, I. T. (2002). Principal Component Analysis (2nd ed.). Springer. The definitive monograph on PCA theory and applications.
Arthur, D., & Vassilvitskii, S. (2007). k-means++: The advantages of careful seeding. Proceedings of the ACM-SIAM Symposium on Discrete Algorithms, 1027-1035. The initialization strategy scikit-learn's
KMeansuses by default.Ester, M., Kriegel, H.-P., Sander, J., & Xu, X. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of KDD, 226-231. The original DBSCAN paper.
scikit-learn Developers. scikit-learn
clusteranddecompositiondocumentation. https://scikit-learn.org/stable/modules/clustering.html