Statistical Basis of AI/ML — Advanced Scientific Python
Chapter 12 — Efficient Computation and Exploratory Analysis at Scale
Why This Chapter?
From Chapter 2 onward you have been writing NumPy and Pandas code that worked correctly. But as datasets grow — from hundreds of rows to hundreds of thousands, from a handful of features to thousands — "working correctly" is no longer sufficient. A nested Python loop over 100 000 samples takes seconds; the equivalent NumPy broadcast takes milliseconds. The difference between interactive analysis and a production training pipeline often comes down to knowing which operations are vectorisable and how to express them.
Chapter 10 showed you the mathematics of linear algebra. Chapter 11 showed you the mathematics of calculus. This chapter closes the gap between the mathematics and the code: how do you express matrix operations, aggregations, statistical tests, and dimensionality reductions in Python in a way that is simultaneously correct, readable, and fast?
The four skills built here are universal in AI/ML engineering. Vectorisation eliminates Python loops by pushing work into compiled C/Fortran kernels — the foundation of every neural network forward pass. Advanced Pandas aggregations turn raw tabular data into per-class summaries and rolling-window smoothed time series — the daily bread of feature engineering. Hypothesis testing on grouped data quantifies whether observed differences between classes are statistically significant or merely noise. And PCA + t-SNE give you the visual tools to understand the geometry of high-dimensional datasets before you model them — a diagnostic step that prevents countless wasted training runs.
$$\boxed{\text{Loop}} \;\xrightarrow{\text{vectorise}}\; \boxed{\text{Array Ops}} \;\xrightarrow{\text{aggregate}}\; \boxed{\text{Grouped Stats}} \;\xrightarrow{\text{reduce}}\; \boxed{\text{PCA / t-SNE}}$$
What You Will Learn
| $\S$ | Topic | Description |
|---|---|---|
| 12.0 | Setup | Imports & configuration |
| 12.1 | Vectorisation and Timing | Loop vs. NumPy broadcast; profiling; pairwise distance comparison |
| 12.2 | Advanced Pandas Aggregations | groupby().agg(), rolling windows, rank-based transforms |
| 12.3 | Statistical Tests on Grouped Data | One-way ANOVA and Welch $t$-test on class features |
| 12.4 | PCA and t-SNE Visualisation | Full pipeline: scale → PCA → t-SNE on Wisconsin BC dataset |
| 12.5 | Summary & Quick Reference | Performance patterns, API cheat sheet |
| 12.6 | References | Authoritative sources |
Section 12.0 — Setup: Imports & Configuration
This chapter uses the full scientific Python stack. The time module enables
micro-benchmarks; sklearn provides the PCA, t-SNE, and scaler implementations
we will examine under the hood.
| Library | Role in This Chapter |
|---|---|
| NumPy | Vectorised array operations, broadcasting, timing |
| Pandas | groupby, agg, rolling, rank — advanced tabular analysis |
| Matplotlib / Seaborn | Timing bar charts, scatter plots, density overlays |
SciPy stats |
f_oneway (ANOVA), ttest_ind (Welch $t$-test) |
| scikit-learn | StandardScaler, PCA, TSNE — reference implementations |
| Plotly Express | Interactive 3-D PCA scatter |
time |
Wall-clock micro-benchmarks via perf_counter |
# === Section 12.0: Imports & configuration ===
import time
import pathlib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
import plotly.express as px
rng = np.random.default_rng(42)
sns.set_theme(style="whitegrid", palette="muted")
print("NumPy", np.__version__, "| scikit-learn ready")
Section 12.1 — Vectorisation and Timing
Python's for loop executes one operation at a time through the interpreter —
each iteration involves type-checking, reference counting, and bytecode dispatch.
NumPy's array operations bypass all of that: they call compiled C and Fortran
kernels that operate on contiguous memory blocks. The theoretical speedup for
simple element-wise arithmetic is determined by the ratio of Python overhead to
actual flop time, which can easily exceed 100×.
The fundamental mechanism is broadcasting: NumPy's rules for implicitly expanding dimensions when operating on arrays of different shapes. For a scalar-valued function $f(\mathbf{a}, \mathbf{b}) = \|\mathbf{a} - \mathbf{b}\|^2$ evaluated over all $n^2$ pairs, the broadcasting idiom is:
$$\boxed{D_{ij} = \sum_k (A_{ik} - A_{jk})^2 \;\equiv\; \texttt{A[:,None,:] - A[None,:,:]}}$$
This single expression creates a $(n, n, d)$ intermediate and reduces along the last axis — 10–100× faster than a double Python loop for $n = 200$, $d = 4$. In deep learning, every batched matrix multiply and every attention computation uses broadcasting under the hood.
The code below measures three cases: (1) element-wise squared-sum in a loop vs. NumPy, (2) pairwise squared distance in a double loop vs. broadcasting.
# === Section 12.1: Loop vs vectorised element-wise sum of squares ===
N = 50_000
v = rng.standard_normal(N)
# Python loop
t0 = time.perf_counter()
total_loop = sum(xi**2 for xi in v)
t_loop = time.perf_counter() - t0
# NumPy vectorised
t0 = time.perf_counter()
total_vec = (v**2).sum()
t_vec = time.perf_counter() - t0
print(f"Python loop: {t_loop*1e3:.2f} ms result={total_loop:.4f}")
print(f"NumPy vector: {t_vec*1e3:.2f} ms result={total_vec:.4f}")
print(f"Speedup: {t_loop/t_vec:.1f}×")
# === Section 12.1 (cont.): Pairwise squared distances — loop vs broadcast ===
A = rng.standard_normal((200, 4))
# Naive double loop
t0 = time.perf_counter()
D_loop = np.zeros((200, 200))
for i in range(200):
for j in range(200):
D_loop[i, j] = np.sum((A[i] - A[j])**2)
t_loop2 = time.perf_counter() - t0
# Broadcasting: (200, 1, 4) - (1, 200, 4) → (200, 200, 4) → sum → (200, 200)
t0 = time.perf_counter()
D_bc = np.sum((A[:, None, :] - A[None, :, :])**2, axis=-1)
t_bc = time.perf_counter() - t0
print(f"Loop (200×200): {t_loop2*1e3:.1f} ms")
print(f"Broadcast: {t_bc*1e3:.1f} ms speedup={t_loop2/t_bc:.1f}×")
print(f"Max diff: {np.max(np.abs(D_loop - D_bc)):.2e} (should be ~0)")
Section 12.2 — Advanced Pandas Aggregations
Pandas groupby is the workhorse of per-class feature engineering. The pattern
df.groupby(label_col)[feature_cols].agg(["mean", "std", "median"])
computes multiple summary statistics simultaneously in a single pass — far more
efficient than calling .mean(), .std(), .median() separately. The result
is a DataFrame with a MultiIndex on the columns.
For time-ordered or sorted data, the rolling window transform applies an aggregation function over a sliding window of fixed size:
$$\boxed{\hat{\mu}_t = \frac{1}{W}\sum_{k=t-W+1}^{t} x_k}$$
Rolling means and rolling standard deviations are fundamental in financial
time-series analysis, sensor-data smoothing, and online learning. The center=True
option centres the window so that the smoothed value at time $t$ uses data
$[t-W/2, \, t+W/2]$ — useful for offline analysis but not valid for real-time
deployment (it uses future data).
Below we load the UCI Wine dataset, compute class-level aggregations, and plot a rolling mean with $\pm 1$ standard deviation band for the first numeric feature.
# === Section 12.2: Load UCI Wine dataset ===
data_root = pathlib.Path("../../data")
df_wine = pd.read_csv(data_root / "wine_data.csv")
print("Shape:", df_wine.shape)
print("Columns:", df_wine.columns.tolist()[:8], "...")
# === Section 12.2 (cont.): Grouped aggregations ===
label_col = "target" if "target" in df_wine.columns else df_wine.columns[-1]
feat_cols = [c for c in df_wine.select_dtypes("number").columns if c != label_col]
agg = (df_wine
.groupby(label_col)[feat_cols[:5]]
.agg(["mean", "std", "median"])
.round(3))
print("Per-class aggregations (first 5 features):")
print(agg.to_string())
# === Section 12.2 (cont.): Rolling statistics on sorted feature ===
sorted_vals = df_wine[feat_cols[0]].sort_values().reset_index(drop=True)
roll_mean = sorted_vals.rolling(window=20, center=True).mean()
roll_std = sorted_vals.rolling(window=20, center=True).std()
fig, ax = plt.subplots(figsize=(10, 4))
ax.plot(sorted_vals.values, alpha=0.35, lw=1, label="raw values")
ax.plot(roll_mean, lw=2, label="rolling mean (w=20)")
ax.fill_between(roll_mean.index,
roll_mean - roll_std,
roll_mean + roll_std,
alpha=0.25, label="±1 std band")
ax.legend(); ax.set_title(f"Rolling statistics — {feat_cols[0]}")
ax.set_xlabel("sorted index"); ax.set_ylabel("value")
plt.tight_layout(); plt.show()
Section 12.3 — Statistical Tests on Grouped Data
In Chapter 5 you learned one-sample and two-sample $t$-tests. In practice, many datasets have $k \geq 3$ class labels, and the correct test is the one-way ANOVA (Analysis of Variance), which tests:
$$H_0: \mu_1 = \mu_2 = \cdots = \mu_k \quad\text{vs.}\quad H_a: \text{at least one mean differs}$$
The $F$-statistic compares between-group variance to within-group variance:
$$\boxed{F = \frac{\text{MS}_{\text{between}}}{\text{MS}_{\text{within}}} = \frac{\sum_{j} n_j(\bar{x}_j - \bar{x})^2 / (k-1)}{\sum_{j}\sum_{i}(x_{ij} - \bar{x}_j)^2 / (n-k)}}$$
A large $F$ (small $p$-value) means the class label explains a meaningful fraction of the total variance in the feature. For binary labels, ANOVA reduces to the two-sample Welch $t$-test.
Applying this test across all features simultaneously is a fast (though statistically informal — it ignores multiple-comparison correction) way to identify which features discriminate most strongly between classes — a useful first pass before fitting a model.
# === Section 12.3: ANOVA / Welch t-test per feature ===
class_labels = df_wine[label_col].unique()
n_classes = len(class_labels)
print(f"Testing feature '{feat_cols[0]}' across {n_classes} classes:\n")
groups = [df_wine[df_wine[label_col] == g][feat_cols[0]].dropna().values
for g in class_labels]
if n_classes == 2:
stat, p = stats.ttest_ind(*groups, equal_var=False)
test_name = "Welch t-test"
else:
stat, p = stats.f_oneway(*groups)
test_name = "One-way ANOVA F"
alpha = 0.05
verdict = "Significant ✓" if p < alpha else "Not significant"
print(f"{test_name} = {stat:.3f} p = {p:.4e}")
print(f"Conclusion: {verdict} at α={alpha}")
# === Section 12.3 (cont.): F-statistics for all numeric features ===
results = []
for feat in feat_cols:
gs = [df_wine[df_wine[label_col] == g][feat].dropna().values for g in class_labels]
if n_classes == 2:
f_stat, pv = stats.ttest_ind(*gs, equal_var=False)
f_stat = f_stat**2 # t² ≈ F for equal variance
else:
f_stat, pv = stats.f_oneway(*gs)
results.append({"feature": feat, "F": f_stat, "p": pv, "sig": pv < 0.05})
df_res = pd.DataFrame(results).sort_values("F", ascending=False)
fig, ax = plt.subplots(figsize=(10, 4))
colors = ["tomato" if s else "steelblue" for s in df_res["sig"]]
ax.barh(df_res["feature"], df_res["F"], color=colors)
ax.set_xlabel("F-statistic (log scale)"); ax.set_xscale("log")
ax.set_title(f"Feature discriminability (red = significant at α=0.05, {n_classes} classes)")
plt.tight_layout(); plt.show()
print(f"\nTop 5 discriminating features:\n{df_res[['feature','F','p']].head().to_string(index=False)}")
Section 12.4 — PCA and t-SNE on the Wisconsin Breast Cancer Dataset
High-dimensional data cannot be inspected directly. Dimensionality reduction projects it onto 2 or 3 dimensions while preserving the most important structure. Two complementary methods are used here:
PCA (Principal Component Analysis) is a linear projection that finds the orthogonal directions of maximum variance. It is deterministic, fast, and globally consistent — the same run on the same data always gives the same result. The fraction of variance explained by the top-$k$ components is:
$$\boxed{\text{EVR}_k = \frac{\sum_{i=1}^k \sigma_i^2}{\sum_{i=1}^n \sigma_i^2}}$$
A scree plot of EVR versus component index tells you how many components to keep.
t-SNE (t-distributed Stochastic Neighbour Embedding, van der Maaten & Hinton 2008) is a non-linear method that preserves local neighbourhood structure. It places each point in a 2-D map such that similar points (in the original high-dimensional space) have high probability of being neighbours in the map. t-SNE is excellent for revealing cluster structure invisible to PCA, but is stochastic, scale-free, and computationally $O(n^2)$ (or $O(n \log n)$ with the Barnes–Hut approximation).
Important preprocessing: Both PCA and t-SNE are sensitive to feature scale.
Always standardise (zero mean, unit variance) before applying them. The
StandardScaler from scikit-learn does this in one step.
# === Section 12.4: Load and preprocess Wisconsin BC dataset ===
bc_path = data_root / "wisc_bc_data.csv"
df_bc = pd.read_csv(bc_path)
print("Shape:", df_bc.shape)
id_col = [c for c in df_bc.columns if "id" in c.lower()]
diag_col = [c for c in df_bc.columns if "diag" in c.lower() or c == "diagnosis"]
drop_cols = id_col + diag_col
label_bc = diag_col[0] if diag_col else df_bc.columns[1]
X_bc = df_bc.drop(columns=drop_cols, errors="ignore").select_dtypes("number").values
y_bc = df_bc[label_bc].values
print(f"Features: {X_bc.shape[1]} Classes: {np.unique(y_bc)}")
sc = StandardScaler()
X_sc = sc.fit_transform(X_bc)
# === Section 12.4 (cont.): PCA — scree plot + 2-D scatter ===
pca = PCA(n_components=10, random_state=42)
Z_pca = pca.fit_transform(X_sc)
evr = pca.explained_variance_ratio_
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
# Scree
axes[0].bar(range(1, 11), evr * 100, color="steelblue", edgecolor="k")
axes[0].plot(range(1, 11), np.cumsum(evr) * 100, "o-", color="tomato", label="cumulative")
axes[0].set_xlabel("Principal Component"); axes[0].set_ylabel("% Variance Explained")
axes[0].set_title("Scree Plot — Wisconsin BC"); axes[0].legend()
# 2-D scatter
palette = {"M": "tomato", "B": "steelblue"}
for lab, col in palette.items():
mask = y_bc == lab
axes[1].scatter(Z_pca[mask, 0], Z_pca[mask, 1], label=lab, alpha=0.5, s=18, c=col)
axes[1].set_xlabel(f"PC1 ({evr[0]*100:.1f}%)")
axes[1].set_ylabel(f"PC2 ({evr[1]*100:.1f}%)")
axes[1].set_title("PCA 2-D — Wisconsin BC"); axes[1].legend()
plt.tight_layout(); plt.show()
print(f"Top-2 PCs explain {np.sum(evr[:2])*100:.1f}% of variance")
# === Section 12.4 (cont.): Interactive 3-D PCA ===
fig3d = px.scatter_3d(
x=Z_pca[:, 0], y=Z_pca[:, 1], z=Z_pca[:, 2],
color=y_bc,
title="PCA 3-D — Wisconsin BC Dataset",
labels={"x": f"PC1 ({evr[0]*100:.1f}%)",
"y": f"PC2 ({evr[1]*100:.1f}%)",
"z": f"PC3 ({evr[2]*100:.1f}%)"},
color_discrete_map={"M": "tomato", "B": "steelblue"},
)
fig3d.show()
# === Section 12.4 (cont.): t-SNE on a 320-sample subset ===
idx = rng.choice(len(X_sc), 320, replace=False)
tsne = TSNE(n_components=2, perplexity=30, random_state=42, max_iter=1000)
Z_tsne = tsne.fit_transform(X_sc[idx])
fig, ax = plt.subplots(figsize=(7, 5))
for lab, col in palette.items():
mask = y_bc[idx] == lab
ax.scatter(Z_tsne[mask, 0], Z_tsne[mask, 1], label=lab, alpha=0.65, s=20, c=col)
ax.set_title("t-SNE 2-D (n=320) — Wisconsin BC Dataset")
ax.set_xlabel("t-SNE dim 1"); ax.set_ylabel("t-SNE dim 2")
ax.legend(); plt.tight_layout(); plt.show()
Section 12.5 — Summary & Quick Reference
Key Patterns¶
| Task | Avoid | Use instead |
|---|---|---|
| Element-wise sum | for x in arr: total += x |
arr.sum() |
| Row norms | for i: np.linalg.norm(A[i]) |
np.linalg.norm(A, axis=1) |
| Pairwise distances | double loop | A[:, None, :] - A[None, :, :] |
| Per-class stats | separate .mean(), .std() |
.groupby().agg([...]) |
| Rolling average | loop with sliding index | Series.rolling(w).mean() |
| Group test | manual F formula | scipy.stats.f_oneway(*groups) |
| Dimension reduction | raw scatter in 30-D | PCA → t-SNE pipeline |
API Cheat Sheet¶
# Vectorisation
norms = np.linalg.norm(X, axis=1, keepdims=True)
D_sq = np.sum((A[:, None, :] - A[None, :, :])**2, axis=-1)
# Groupby aggregation
df.groupby("class")[feats].agg(["mean", "std", "median"])
# Rolling
Series.rolling(window=20, center=True).mean()
# ANOVA
stat, p = scipy.stats.f_oneway(group1, group2, group3)
# PCA + t-SNE pipeline
scaler = StandardScaler()
pca = PCA(n_components=50, random_state=42)
tsne = TSNE(n_components=2, perplexity=30, random_state=42)
Z = tsne.fit_transform(pca.fit_transform(scaler.fit_transform(X)))
What's Next?¶
$$\boxed{\text{Ch 12: Scientific Python}} \;\to\; \boxed{\text{Ch 13: Probability \& Info Theory}} \;\to\; \boxed{\text{Ch 14: Neural Networks}}$$
Chapter 13 returns to the mathematical foundations: the probabilistic framework that underpins every classifier output — entropy, cross-entropy, KL divergence, and softmax — the building blocks of neural network loss functions.
References¶
VanderPlas, J. (2023). Python Data Science Handbook (2nd ed.). O'Reilly Media. Comprehensive coverage of NumPy broadcasting, Pandas aggregations, and scikit-learn. https://jakevdp.github.io/PythonDataScienceHandbook/
McKinney, W. (2022). Python for Data Analysis (3rd ed.). O'Reilly Media. The canonical Pandas reference; Chapters 9–10 cover
groupbyand aggregation in depth.Virtanen, P., et al. (2020). SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17, 261–272. https://doi.org/10.1038/s41592-019-0686-2
van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9, 2579–2605. Original t-SNE paper; explains the perplexity hyperparameter and gradient derivation. http://jmlr.org/papers/v9/vandermaaten08a.html
Harris, C. R., et al. (2020). Array programming with NumPy. Nature, 585, 357–362. https://doi.org/10.1038/s41586-020-2649-2 The definitive NumPy reference paper.
scikit-learn developers. User Guide: Decomposition (PCA) and Manifold learning (t-SNE). https://scikit-learn.org/stable/modules/decomposition.html https://scikit-learn.org/stable/modules/manifold.html
NumPy Development Team. NumPy Broadcasting guide. https://numpy.org/doc/stable/user/basics.broadcasting.html
Pandas Development Team. Group By: split-apply-combine. https://pandas.pydata.org/docs/user_guide/groupby.html