Home / About Us / Volume I Contents / Ch 12: Advanced Scientific Python

Statistical Basis of AI/ML — Advanced Scientific Python

Chapter 12 — Efficient Computation and Exploratory Analysis at Scale


Why This Chapter?

From Chapter 2 onward you have been writing NumPy and Pandas code that worked correctly. But as datasets grow — from hundreds of rows to hundreds of thousands, from a handful of features to thousands — "working correctly" is no longer sufficient. A nested Python loop over 100 000 samples takes seconds; the equivalent NumPy broadcast takes milliseconds. The difference between interactive analysis and a production training pipeline often comes down to knowing which operations are vectorisable and how to express them.

Chapter 10 showed you the mathematics of linear algebra. Chapter 11 showed you the mathematics of calculus. This chapter closes the gap between the mathematics and the code: how do you express matrix operations, aggregations, statistical tests, and dimensionality reductions in Python in a way that is simultaneously correct, readable, and fast?

The four skills built here are universal in AI/ML engineering. Vectorisation eliminates Python loops by pushing work into compiled C/Fortran kernels — the foundation of every neural network forward pass. Advanced Pandas aggregations turn raw tabular data into per-class summaries and rolling-window smoothed time series — the daily bread of feature engineering. Hypothesis testing on grouped data quantifies whether observed differences between classes are statistically significant or merely noise. And PCA + t-SNE give you the visual tools to understand the geometry of high-dimensional datasets before you model them — a diagnostic step that prevents countless wasted training runs.

$$\boxed{\text{Loop}} \;\xrightarrow{\text{vectorise}}\; \boxed{\text{Array Ops}} \;\xrightarrow{\text{aggregate}}\; \boxed{\text{Grouped Stats}} \;\xrightarrow{\text{reduce}}\; \boxed{\text{PCA / t-SNE}}$$

What You Will Learn

$\S$ Topic Description
12.0 Setup Imports & configuration
12.1 Vectorisation and Timing Loop vs. NumPy broadcast; profiling; pairwise distance comparison
12.2 Advanced Pandas Aggregations groupby().agg(), rolling windows, rank-based transforms
12.3 Statistical Tests on Grouped Data One-way ANOVA and Welch $t$-test on class features
12.4 PCA and t-SNE Visualisation Full pipeline: scale → PCA → t-SNE on Wisconsin BC dataset
12.5 Summary & Quick Reference Performance patterns, API cheat sheet
12.6 References Authoritative sources

Section 12.0 — Setup: Imports & Configuration

This chapter uses the full scientific Python stack. The time module enables micro-benchmarks; sklearn provides the PCA, t-SNE, and scaler implementations we will examine under the hood.

Library Role in This Chapter
NumPy Vectorised array operations, broadcasting, timing
Pandas groupby, agg, rolling, rank — advanced tabular analysis
Matplotlib / Seaborn Timing bar charts, scatter plots, density overlays
SciPy stats f_oneway (ANOVA), ttest_ind (Welch $t$-test)
scikit-learn StandardScaler, PCA, TSNE — reference implementations
Plotly Express Interactive 3-D PCA scatter
time Wall-clock micro-benchmarks via perf_counter
In [1]:
# === Section 12.0: Imports & configuration ===
import time
import pathlib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
import plotly.express as px

rng = np.random.default_rng(42)
sns.set_theme(style="whitegrid", palette="muted")
print("NumPy", np.__version__, "| scikit-learn ready")
NumPy 2.3.5 | scikit-learn ready

Section 12.1 — Vectorisation and Timing

Python's for loop executes one operation at a time through the interpreter — each iteration involves type-checking, reference counting, and bytecode dispatch. NumPy's array operations bypass all of that: they call compiled C and Fortran kernels that operate on contiguous memory blocks. The theoretical speedup for simple element-wise arithmetic is determined by the ratio of Python overhead to actual flop time, which can easily exceed 100×.

The fundamental mechanism is broadcasting: NumPy's rules for implicitly expanding dimensions when operating on arrays of different shapes. For a scalar-valued function $f(\mathbf{a}, \mathbf{b}) = \|\mathbf{a} - \mathbf{b}\|^2$ evaluated over all $n^2$ pairs, the broadcasting idiom is:

$$\boxed{D_{ij} = \sum_k (A_{ik} - A_{jk})^2 \;\equiv\; \texttt{A[:,None,:] - A[None,:,:]}}$$

This single expression creates a $(n, n, d)$ intermediate and reduces along the last axis — 10–100× faster than a double Python loop for $n = 200$, $d = 4$. In deep learning, every batched matrix multiply and every attention computation uses broadcasting under the hood.

The code below measures three cases: (1) element-wise squared-sum in a loop vs. NumPy, (2) pairwise squared distance in a double loop vs. broadcasting.

In [2]:
# === Section 12.1: Loop vs vectorised element-wise sum of squares ===
N = 50_000
v = rng.standard_normal(N)

# Python loop
t0 = time.perf_counter()
total_loop = sum(xi**2 for xi in v)
t_loop = time.perf_counter() - t0

# NumPy vectorised
t0 = time.perf_counter()
total_vec = (v**2).sum()
t_vec = time.perf_counter() - t0

print(f"Python loop:   {t_loop*1e3:.2f} ms  result={total_loop:.4f}")
print(f"NumPy vector:  {t_vec*1e3:.2f} ms  result={total_vec:.4f}")
print(f"Speedup:       {t_loop/t_vec:.1f}×")
Python loop:   4.86 ms  result=50261.7001
NumPy vector:  0.11 ms  result=50261.7001
Speedup:       43.1×
In [3]:
# === Section 12.1 (cont.): Pairwise squared distances — loop vs broadcast ===
A = rng.standard_normal((200, 4))

# Naive double loop
t0 = time.perf_counter()
D_loop = np.zeros((200, 200))
for i in range(200):
    for j in range(200):
        D_loop[i, j] = np.sum((A[i] - A[j])**2)
t_loop2 = time.perf_counter() - t0

# Broadcasting: (200, 1, 4) - (1, 200, 4) → (200, 200, 4) → sum → (200, 200)
t0 = time.perf_counter()
D_bc = np.sum((A[:, None, :] - A[None, :, :])**2, axis=-1)
t_bc = time.perf_counter() - t0

print(f"Loop (200×200): {t_loop2*1e3:.1f} ms")
print(f"Broadcast:      {t_bc*1e3:.1f} ms   speedup={t_loop2/t_bc:.1f}×")
print(f"Max diff:       {np.max(np.abs(D_loop - D_bc)):.2e}  (should be ~0)")
Loop (200×200): 84.2 ms
Broadcast:      0.8 ms   speedup=103.6×
Max diff:       0.00e+00  (should be ~0)

Section 12.2 — Advanced Pandas Aggregations

Pandas groupby is the workhorse of per-class feature engineering. The pattern

df.groupby(label_col)[feature_cols].agg(["mean", "std", "median"])

computes multiple summary statistics simultaneously in a single pass — far more efficient than calling .mean(), .std(), .median() separately. The result is a DataFrame with a MultiIndex on the columns.

For time-ordered or sorted data, the rolling window transform applies an aggregation function over a sliding window of fixed size:

$$\boxed{\hat{\mu}_t = \frac{1}{W}\sum_{k=t-W+1}^{t} x_k}$$

Rolling means and rolling standard deviations are fundamental in financial time-series analysis, sensor-data smoothing, and online learning. The center=True option centres the window so that the smoothed value at time $t$ uses data $[t-W/2, \, t+W/2]$ — useful for offline analysis but not valid for real-time deployment (it uses future data).

Below we load the UCI Wine dataset, compute class-level aggregations, and plot a rolling mean with $\pm 1$ standard deviation band for the first numeric feature.

In [4]:
# === Section 12.2: Load UCI Wine dataset ===
data_root = pathlib.Path("../../data")
df_wine   = pd.read_csv(data_root / "wine_data.csv")
print("Shape:", df_wine.shape)
print("Columns:", df_wine.columns.tolist()[:8], "...")
Shape: (4898, 12)
Columns: ['fixed acidity', 'volatile acidity', 'citric acid', 'residual sugar', 'chlorides', 'free sulfur dioxide', 'total sulfur dioxide', 'density'] ...
In [5]:
# === Section 12.2 (cont.): Grouped aggregations ===
label_col = "target" if "target" in df_wine.columns else df_wine.columns[-1]
feat_cols  = [c for c in df_wine.select_dtypes("number").columns if c != label_col]

agg = (df_wine
       .groupby(label_col)[feat_cols[:5]]
       .agg(["mean", "std", "median"])
       .round(3))
print("Per-class aggregations (first 5 features):")
print(agg.to_string())
Per-class aggregations (first 5 features):
        fixed acidity               volatile acidity               citric acid               residual sugar               chlorides              
                 mean    std median             mean    std median        mean    std median           mean    std median      mean    std median
quality                                                                                                                                          
3               7.600  1.725    7.3            0.333  0.141   0.26       0.336  0.081  0.345          6.392  5.317   4.60     0.054  0.046  0.041
4               7.129  1.081    6.9            0.381  0.173   0.32       0.304  0.164  0.290          4.628  4.164   2.50     0.050  0.026  0.046
5               6.934  0.838    6.8            0.302  0.100   0.28       0.338  0.141  0.320          7.335  5.328   7.00     0.052  0.026  0.047
6               6.838  0.837    6.8            0.261  0.088   0.25       0.338  0.119  0.320          6.442  5.167   5.30     0.045  0.020  0.043
7               6.735  0.756    6.7            0.263  0.091   0.25       0.326  0.079  0.310          5.186  4.298   3.65     0.038  0.011  0.037
8               6.657  0.819    6.8            0.277  0.108   0.26       0.327  0.085  0.320          5.671  4.263   4.30     0.038  0.013  0.036
9               7.420  0.983    7.1            0.298  0.058   0.27       0.386  0.082  0.360          4.120  3.759   2.20     0.027  0.007  0.031
In [6]:
# === Section 12.2 (cont.): Rolling statistics on sorted feature ===
sorted_vals  = df_wine[feat_cols[0]].sort_values().reset_index(drop=True)
roll_mean    = sorted_vals.rolling(window=20, center=True).mean()
roll_std     = sorted_vals.rolling(window=20, center=True).std()

fig, ax = plt.subplots(figsize=(10, 4))
ax.plot(sorted_vals.values, alpha=0.35, lw=1, label="raw values")
ax.plot(roll_mean, lw=2,   label="rolling mean (w=20)")
ax.fill_between(roll_mean.index,
                roll_mean - roll_std,
                roll_mean + roll_std,
                alpha=0.25, label="±1 std band")
ax.legend(); ax.set_title(f"Rolling statistics — {feat_cols[0]}")
ax.set_xlabel("sorted index"); ax.set_ylabel("value")
plt.tight_layout(); plt.show()
No description has been provided for this image

Section 12.3 — Statistical Tests on Grouped Data

In Chapter 5 you learned one-sample and two-sample $t$-tests. In practice, many datasets have $k \geq 3$ class labels, and the correct test is the one-way ANOVA (Analysis of Variance), which tests:

$$H_0: \mu_1 = \mu_2 = \cdots = \mu_k \quad\text{vs.}\quad H_a: \text{at least one mean differs}$$

The $F$-statistic compares between-group variance to within-group variance:

$$\boxed{F = \frac{\text{MS}_{\text{between}}}{\text{MS}_{\text{within}}} = \frac{\sum_{j} n_j(\bar{x}_j - \bar{x})^2 / (k-1)}{\sum_{j}\sum_{i}(x_{ij} - \bar{x}_j)^2 / (n-k)}}$$

A large $F$ (small $p$-value) means the class label explains a meaningful fraction of the total variance in the feature. For binary labels, ANOVA reduces to the two-sample Welch $t$-test.

Applying this test across all features simultaneously is a fast (though statistically informal — it ignores multiple-comparison correction) way to identify which features discriminate most strongly between classes — a useful first pass before fitting a model.

In [7]:
# === Section 12.3: ANOVA / Welch t-test per feature ===
class_labels = df_wine[label_col].unique()
n_classes    = len(class_labels)

print(f"Testing feature '{feat_cols[0]}' across {n_classes} classes:\n")
groups = [df_wine[df_wine[label_col] == g][feat_cols[0]].dropna().values
          for g in class_labels]

if n_classes == 2:
    stat, p = stats.ttest_ind(*groups, equal_var=False)
    test_name = "Welch t-test"
else:
    stat, p = stats.f_oneway(*groups)
    test_name = "One-way ANOVA  F"

alpha = 0.05
verdict = "Significant ✓" if p < alpha else "Not significant"
print(f"{test_name} = {stat:.3f}   p = {p:.4e}")
print(f"Conclusion: {verdict} at α={alpha}")
Testing feature 'fixed acidity' across 7 classes:

One-way ANOVA  F = 12.895   p = 1.6385e-14
Conclusion: Significant ✓ at α=0.05
In [8]:
# === Section 12.3 (cont.): F-statistics for all numeric features ===
results = []
for feat in feat_cols:
    gs = [df_wine[df_wine[label_col] == g][feat].dropna().values for g in class_labels]
    if n_classes == 2:
        f_stat, pv = stats.ttest_ind(*gs, equal_var=False)
        f_stat = f_stat**2   # t² ≈ F for equal variance
    else:
        f_stat, pv = stats.f_oneway(*gs)
    results.append({"feature": feat, "F": f_stat, "p": pv, "sig": pv < 0.05})

df_res = pd.DataFrame(results).sort_values("F", ascending=False)

fig, ax = plt.subplots(figsize=(10, 4))
colors = ["tomato" if s else "steelblue" for s in df_res["sig"]]
ax.barh(df_res["feature"], df_res["F"], color=colors)
ax.set_xlabel("F-statistic (log scale)"); ax.set_xscale("log")
ax.set_title(f"Feature discriminability (red = significant at α=0.05,  {n_classes} classes)")
plt.tight_layout(); plt.show()
print(f"\nTop 5 discriminating features:\n{df_res[['feature','F','p']].head().to_string(index=False)}")
No description has been provided for this image
Top 5 discriminating features:
             feature          F             p
             alcohol 229.734772 2.925427e-259
             density 105.856434 8.499687e-126
    volatile acidity  61.916739  2.660979e-74
total sulfur dioxide  45.200949  4.044013e-54
           chlorides  42.472252  8.504480e-51

Section 12.4 — PCA and t-SNE on the Wisconsin Breast Cancer Dataset

High-dimensional data cannot be inspected directly. Dimensionality reduction projects it onto 2 or 3 dimensions while preserving the most important structure. Two complementary methods are used here:

PCA (Principal Component Analysis) is a linear projection that finds the orthogonal directions of maximum variance. It is deterministic, fast, and globally consistent — the same run on the same data always gives the same result. The fraction of variance explained by the top-$k$ components is:

$$\boxed{\text{EVR}_k = \frac{\sum_{i=1}^k \sigma_i^2}{\sum_{i=1}^n \sigma_i^2}}$$

A scree plot of EVR versus component index tells you how many components to keep.

t-SNE (t-distributed Stochastic Neighbour Embedding, van der Maaten & Hinton 2008) is a non-linear method that preserves local neighbourhood structure. It places each point in a 2-D map such that similar points (in the original high-dimensional space) have high probability of being neighbours in the map. t-SNE is excellent for revealing cluster structure invisible to PCA, but is stochastic, scale-free, and computationally $O(n^2)$ (or $O(n \log n)$ with the Barnes–Hut approximation).

Important preprocessing: Both PCA and t-SNE are sensitive to feature scale. Always standardise (zero mean, unit variance) before applying them. The StandardScaler from scikit-learn does this in one step.

In [9]:
# === Section 12.4: Load and preprocess Wisconsin BC dataset ===
bc_path = data_root / "wisc_bc_data.csv"
df_bc   = pd.read_csv(bc_path)
print("Shape:", df_bc.shape)

id_col   = [c for c in df_bc.columns if "id"   in c.lower()]
diag_col = [c for c in df_bc.columns if "diag" in c.lower() or c == "diagnosis"]
drop_cols = id_col + diag_col
label_bc  = diag_col[0] if diag_col else df_bc.columns[1]

X_bc = df_bc.drop(columns=drop_cols, errors="ignore").select_dtypes("number").values
y_bc = df_bc[label_bc].values
print(f"Features: {X_bc.shape[1]}   Classes: {np.unique(y_bc)}")

sc   = StandardScaler()
X_sc = sc.fit_transform(X_bc)
Shape: (569, 31)
Features: 30   Classes: ['B' 'M']
In [10]:
# === Section 12.4 (cont.): PCA — scree plot + 2-D scatter ===
pca  = PCA(n_components=10, random_state=42)
Z_pca = pca.fit_transform(X_sc)
evr   = pca.explained_variance_ratio_

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

# Scree
axes[0].bar(range(1, 11), evr * 100, color="steelblue", edgecolor="k")
axes[0].plot(range(1, 11), np.cumsum(evr) * 100, "o-", color="tomato", label="cumulative")
axes[0].set_xlabel("Principal Component"); axes[0].set_ylabel("% Variance Explained")
axes[0].set_title("Scree Plot — Wisconsin BC"); axes[0].legend()

# 2-D scatter
palette = {"M": "tomato", "B": "steelblue"}
for lab, col in palette.items():
    mask = y_bc == lab
    axes[1].scatter(Z_pca[mask, 0], Z_pca[mask, 1], label=lab, alpha=0.5, s=18, c=col)
axes[1].set_xlabel(f"PC1 ({evr[0]*100:.1f}%)")
axes[1].set_ylabel(f"PC2 ({evr[1]*100:.1f}%)")
axes[1].set_title("PCA 2-D — Wisconsin BC"); axes[1].legend()
plt.tight_layout(); plt.show()

print(f"Top-2 PCs explain {np.sum(evr[:2])*100:.1f}% of variance")
No description has been provided for this image
Top-2 PCs explain 63.2% of variance
In [11]:
# === Section 12.4 (cont.): Interactive 3-D PCA ===
fig3d = px.scatter_3d(
    x=Z_pca[:, 0], y=Z_pca[:, 1], z=Z_pca[:, 2],
    color=y_bc,
    title="PCA 3-D — Wisconsin BC Dataset",
    labels={"x": f"PC1 ({evr[0]*100:.1f}%)",
            "y": f"PC2 ({evr[1]*100:.1f}%)",
            "z": f"PC3 ({evr[2]*100:.1f}%)"},
    color_discrete_map={"M": "tomato", "B": "steelblue"},
)
fig3d.show()
In [12]:
# === Section 12.4 (cont.): t-SNE on a 320-sample subset ===
idx    = rng.choice(len(X_sc), 320, replace=False)
tsne   = TSNE(n_components=2, perplexity=30, random_state=42, max_iter=1000)
Z_tsne = tsne.fit_transform(X_sc[idx])

fig, ax = plt.subplots(figsize=(7, 5))
for lab, col in palette.items():
    mask = y_bc[idx] == lab
    ax.scatter(Z_tsne[mask, 0], Z_tsne[mask, 1], label=lab, alpha=0.65, s=20, c=col)
ax.set_title("t-SNE 2-D (n=320) — Wisconsin BC Dataset")
ax.set_xlabel("t-SNE dim 1"); ax.set_ylabel("t-SNE dim 2")
ax.legend(); plt.tight_layout(); plt.show()
No description has been provided for this image

Section 12.5 — Summary & Quick Reference

Key Patterns

Task Avoid Use instead
Element-wise sum for x in arr: total += x arr.sum()
Row norms for i: np.linalg.norm(A[i]) np.linalg.norm(A, axis=1)
Pairwise distances double loop A[:, None, :] - A[None, :, :]
Per-class stats separate .mean(), .std() .groupby().agg([...])
Rolling average loop with sliding index Series.rolling(w).mean()
Group test manual F formula scipy.stats.f_oneway(*groups)
Dimension reduction raw scatter in 30-D PCA → t-SNE pipeline

API Cheat Sheet

# Vectorisation
norms = np.linalg.norm(X, axis=1, keepdims=True)
D_sq  = np.sum((A[:, None, :] - A[None, :, :])**2, axis=-1)

# Groupby aggregation
df.groupby("class")[feats].agg(["mean", "std", "median"])

# Rolling
Series.rolling(window=20, center=True).mean()

# ANOVA
stat, p = scipy.stats.f_oneway(group1, group2, group3)

# PCA + t-SNE pipeline
scaler = StandardScaler()
pca    = PCA(n_components=50, random_state=42)
tsne   = TSNE(n_components=2,  perplexity=30, random_state=42)
Z      = tsne.fit_transform(pca.fit_transform(scaler.fit_transform(X)))

What's Next?

$$\boxed{\text{Ch 12: Scientific Python}} \;\to\; \boxed{\text{Ch 13: Probability \& Info Theory}} \;\to\; \boxed{\text{Ch 14: Neural Networks}}$$

Chapter 13 returns to the mathematical foundations: the probabilistic framework that underpins every classifier output — entropy, cross-entropy, KL divergence, and softmax — the building blocks of neural network loss functions.


References

  1. VanderPlas, J. (2023). Python Data Science Handbook (2nd ed.). O'Reilly Media. Comprehensive coverage of NumPy broadcasting, Pandas aggregations, and scikit-learn. https://jakevdp.github.io/PythonDataScienceHandbook/

  2. McKinney, W. (2022). Python for Data Analysis (3rd ed.). O'Reilly Media. The canonical Pandas reference; Chapters 9–10 cover groupby and aggregation in depth.

  3. Virtanen, P., et al. (2020). SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17, 261–272. https://doi.org/10.1038/s41592-019-0686-2

  4. van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9, 2579–2605. Original t-SNE paper; explains the perplexity hyperparameter and gradient derivation. http://jmlr.org/papers/v9/vandermaaten08a.html

  5. Harris, C. R., et al. (2020). Array programming with NumPy. Nature, 585, 357–362. https://doi.org/10.1038/s41586-020-2649-2 The definitive NumPy reference paper.

  6. scikit-learn developers. User Guide: Decomposition (PCA) and Manifold learning (t-SNE). https://scikit-learn.org/stable/modules/decomposition.html https://scikit-learn.org/stable/modules/manifold.html

  7. NumPy Development Team. NumPy Broadcasting guide. https://numpy.org/doc/stable/user/basics.broadcasting.html

  8. Pandas Development Team. Group By: split-apply-combine. https://pandas.pydata.org/docs/user_guide/groupby.html