Statistical Basis of AI/ML — Linear Algebra
Chapter 10 — The Language of Transformations
Why This Chapter?
In Chapter 6 you discovered that ordinary least squares regression finds the coefficient vector $\hat{\boldsymbol{\beta}}$ by solving the normal equations $(\mathbf{X}^T\mathbf{X})\hat{\boldsymbol{\beta}} = \mathbf{X}^T\mathbf{y}$. You used that formula, but the machinery underneath it — matrix inversion, numerical stability, condition numbers — remained implicit. In Chapter 8 you ran PCA and obtained principal components that "rotate" your feature space; those rotations are eigenvectors of the covariance matrix. In Chapter 9 every gradient-boosted tree added an additive correction to a running prediction vector — a sequence of rank-1 updates to a matrix of residuals. You have been doing linear algebra since the first chapter. This chapter makes it explicit.
Every layer in a neural network performs exactly one operation: $\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}$, followed by a non-linearity. Backpropagation is the chain rule applied to a composition of matrix multiplies. Attention in a Transformer is scaled dot-product similarity between query, key, and value matrices. Convolutional layers are structured sparse matrix multiplications. Without a working command of linear algebra, the internals of every modern deep-learning architecture remain a black box.
This chapter builds the four pillars you need: (1) vectors and inner products, which give meaning to similarity; (2) matrix transformations, which give meaning to layers; (3) eigendecomposition, which gives meaning to principal directions; and (4) the singular value decomposition, the most powerful matrix factorisation in applied mathematics, which underlies PCA, collaborative filtering, and weight-matrix compression. The treatment is computational throughout — every theorem is accompanied by a Python demonstration that makes the result tangible.
$$\boxed{\text{Vectors}} \;\xrightarrow{\text{transform}}\; \boxed{\text{Matrices}} \;\xrightarrow{\text{decompose}}\; \boxed{\text{Eigenstructure}} \;\xrightarrow{\text{generalise}}\; \boxed{\text{SVD}}$$
What You Will Learn
| $\S$ | Topic | Description |
|---|---|---|
| 10.0 | Setup | Imports & configuration |
| 10.1 | Vectors: Dot Product, Norm, Cosine Similarity | Inner products, geometric interpretation, batch similarity matrices |
| 10.2 | Matrix Transformations | Rotation, scaling, composition; geometric intuition |
| 10.3 | Eigendecomposition | Definition, computation, PSD matrices, orthogonality |
| 10.4 | SVD and Low-Rank Approximation | Factorisation, energy compaction, PCA via SVD, 3-D visualisation |
| 10.5 | Solving Linear Systems | scipy.linalg.solve, condition number, numerical stability |
| 10.6 | Summary & Quick Reference | Formulas, ML connections, cheat sheet |
| 10.7 | References | Authoritative sources |
Section 10.0 — Setup: Imports & Configuration
Every computational chapter begins with a reproducible setup cell. The libraries below provide vectorised numerical operations (NumPy), a numerically stable linear solver (SciPy), tabular data for the PCA case study (Pandas), and three visualisation layers (Matplotlib, Seaborn, Plotly). Run this cell first.
| Library | Role in This Chapter |
|---|---|
| NumPy | Vectorised array arithmetic, linalg routines: eig, svd, norm, solve |
SciPy linalg |
Numerically stable LU-based solver; condition-number utilities |
| Matplotlib / Seaborn | Geometric scatter plots, similarity heatmaps, SVD rank-comparison grids |
| Plotly Express | Interactive 3-D scatter for PCA visualisation |
| Pandas | Load the UCI Wine dataset for the PCA case study |
# === Section 10.0: Imports & configuration ===
import pathlib
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import linalg
import plotly.express as px
rng = np.random.default_rng(42)
sns.set_theme(style="whitegrid", palette="muted")
print("NumPy", np.__version__, "| SciPy linalg ready")
Section 10.1 — Vectors: Dot Product, Norm, and Cosine Similarity
A vector $\mathbf{a} \in \mathbb{R}^n$ is an ordered tuple of $n$ real numbers. Geometrically it is an arrow from the origin to the point $(a_1, a_2, \ldots, a_n)$ in $n$-dimensional space. The most fundamental operation between two vectors is the dot product (inner product):
$$\boxed{\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_i b_i = \|\mathbf{a}\|\,\|\mathbf{b}\|\cos\theta}$$
where $\theta$ is the angle between the vectors and $\|\cdot\|$ denotes the Euclidean (L2) norm:
$$\boxed{\|\mathbf{a}\| = \sqrt{\sum_{i=1}^{n} a_i^2} = \sqrt{\mathbf{a} \cdot \mathbf{a}}}$$
Dividing the dot product by the product of norms yields the cosine similarity, which lies in $[-1, +1]$:
$$\boxed{\text{sim}(\mathbf{a},\mathbf{b}) = \frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|} = \cos\theta}$$
Cosine similarity is the workhorse of similarity search in modern AI: word
embeddings, sentence encoders, image retrieval systems, and the attention
mechanism all compute cosine (or scaled dot-product) similarity between
high-dimensional vectors. A value of $+1$ means the vectors point in the same
direction (perfectly similar), $0$ means orthogonal (unrelated), and $-1$ means
opposite. In NumPy the dot product is written a @ b.
The code below computes these quantities for two 3-D vectors, then builds a $5 \times 5$ pairwise cosine similarity matrix for a random batch — the same operation that a self-attention head performs over a sequence of token embeddings.
# === Section 10.1: Dot product, norm, cosine similarity ===
a = np.array([1.0, 2.0, 3.0])
b = np.array([4.0, 0.0, -1.0])
dot = a @ b
norm_a = np.linalg.norm(a)
norm_b = np.linalg.norm(b)
cos_sim = dot / (norm_a * norm_b)
angle_deg = np.degrees(np.arccos(np.clip(cos_sim, -1.0, 1.0)))
print(f"dot(a,b) = {dot:.2f}")
print(f"|a| = {norm_a:.4f}")
print(f"|b| = {norm_b:.4f}")
print(f"cosine similarity = {cos_sim:.4f}")
print(f"angle = {angle_deg:.2f}°")
# === Section 10.1 (cont.): Batch pairwise cosine similarity ===
X = rng.standard_normal((5, 4))
norms = np.linalg.norm(X, axis=1, keepdims=True)
X_norm = X / norms # L2-normalise each row
sim_matrix = X_norm @ X_norm.T # (5,5) — all pairwise cosines
fig, ax = plt.subplots(figsize=(5, 4))
sns.heatmap(sim_matrix, annot=True, fmt=".2f", ax=ax,
cmap="coolwarm", vmin=-1, vmax=1, square=True)
ax.set_title("Pairwise Cosine Similarity (5 random 4-D vectors)")
plt.tight_layout()
plt.show()
Section 10.2 — Matrix Transformations
A matrix $\mathbf{M} \in \mathbb{R}^{m \times n}$ defines a linear map $T : \mathbb{R}^n \to \mathbb{R}^m$, $T(\mathbf{x}) = \mathbf{M}\mathbf{x}$. Linear maps preserve addition and scalar multiplication: $T(a\mathbf{u} + b\mathbf{v}) = aT(\mathbf{u}) + bT(\mathbf{v})$. This is exactly the property that makes matrix multiplication the right tool for building composable layers in a neural network.
Two especially important transformations in $\mathbb{R}^2$ are:
Rotation by angle $\theta$: $$\boxed{\mathbf{R}(\theta) = \begin{pmatrix}\cos\theta & -\sin\theta \\ \sin\theta & \cos\theta\end{pmatrix}}$$
Scaling (anisotropic stretch) by factors $s_1, s_2$: $$\boxed{\mathbf{S} = \begin{pmatrix}s_1 & 0 \\ 0 & s_2\end{pmatrix}}$$
Composing both — scale then rotate — gives $\mathbf{M} = \mathbf{R}\,\mathbf{S}$. The order matters: $\mathbf{R}\mathbf{S} \neq \mathbf{S}\mathbf{R}$ in general because matrix multiplication is not commutative. The determinant $\det(\mathbf{M})$ measures the signed area (volume in higher dimensions) scaling factor; if $|\det| > 1$ the transformation expands space, if $|\det| < 1$ it contracts it, and if $\det = 0$ the matrix is singular (non-invertible).
In neural networks, weight matrices $\mathbf{W}$ are general linear maps: they rotate, scale, and project their inputs simultaneously. Understanding how a single matrix deforms a point cloud — as the scatter plot below shows — builds the geometric intuition needed to reason about gradient flow and weight initialisation.
# === Section 10.2: Rotation-and-scale transformation ===
theta = np.pi / 4 # 45 degrees
scale = np.array([2.0, 0.5])
R = np.array([[np.cos(theta), -np.sin(theta)],
[np.sin(theta), np.cos(theta)]])
S = np.diag(scale)
M = R @ S # scale first, then rotate
pts = rng.standard_normal((300, 2))
pts_t = (M @ pts.T).T
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].scatter(pts[:, 0], pts[:, 1], alpha=.4, s=12, color="steelblue")
axes[0].set_title("Original cloud (isotropic Gaussian)")
axes[1].scatter(pts_t[:, 0], pts_t[:, 1], alpha=.4, s=12, color="tomato")
axes[1].set_title(f"After M = R(45°) · S([2, 0.5]) | det(M) = {np.linalg.det(M):.3f}")
for ax in axes:
ax.set_aspect("equal")
ax.axhline(0, c="k", lw=.5)
ax.axvline(0, c="k", lw=.5)
plt.tight_layout()
plt.show()
print(f"det(R)={np.linalg.det(R):.4f} det(S)={np.linalg.det(S):.4f} det(M)={np.linalg.det(M):.4f}")
Section 10.3 — Eigendecomposition
The eigenvalue equation asks: which vectors does a matrix transform purely by scaling (not rotating)?
$$\boxed{\mathbf{A}\,\mathbf{v} = \lambda\,\mathbf{v}}$$
Here $\mathbf{v} \neq \mathbf{0}$ is an eigenvector and $\lambda \in \mathbb{C}$ is the corresponding eigenvalue. A square $n \times n$ matrix has up to $n$ eigenvector–eigenvalue pairs. The eigendecomposition (spectral decomposition) reconstructs the matrix from its eigenpairs:
$$\boxed{\mathbf{A} = \mathbf{V}\,\mathbf{\Lambda}\,\mathbf{V}^{-1}}$$
where the columns of $\mathbf{V}$ are the eigenvectors and $\mathbf{\Lambda}$ is a diagonal matrix of eigenvalues. For symmetric positive semi-definite (PSD) matrices — like covariance matrices — the eigenvectors are orthonormal ($\mathbf{V}^T\mathbf{V} = \mathbf{I}$) and all eigenvalues are $\geq 0$, so:
$$\boxed{\mathbf{A} = \mathbf{V}\,\mathbf{\Lambda}\,\mathbf{V}^T \quad (\text{symmetric PSD})}$$
The trace equals the sum of eigenvalues ($\text{tr}(\mathbf{A}) = \sum_i \lambda_i$) and the determinant equals their product ($\det(\mathbf{A}) = \prod_i \lambda_i$). Eigenvalues near zero signal redundant directions (rank deficiency); very large eigenvalues signal directions of dominant variance — precisely what PCA exploits. In spectral clustering, the eigenvectors of the graph Laplacian define the cluster structure. In quantum mechanics, eigenvalues are observable energy levels — the analogy to ML is not accidental.
# === Section 10.3: Eigendecomposition of a 2×2 matrix ===
A = np.array([[4.0, 2.0],
[1.0, 3.0]])
vals, vecs = np.linalg.eig(A)
print("Eigenvalues :", vals)
print("Eigenvectors (columns):\n", vecs)
print()
for i in range(len(vals)):
residual = np.linalg.norm(A @ vecs[:, i] - vals[i] * vecs[:, i])
print(f" Av{i} - λ{i}v{i} residual = {residual:.2e} (should be ~0)")
print(f"\ntrace(A) = {np.trace(A):.1f} | sum(λ) = {vals.sum():.1f}")
print(f"det(A) = {np.linalg.det(A):.1f} | prod(λ) = {vals.prod():.1f}")
# === Section 10.3 (cont.): Symmetric PSD matrix — orthogonal eigenvectors ===
B = rng.standard_normal((4, 4))
B = B.T @ B # symmetric PSD by construction
vals_b, vecs_b = np.linalg.eigh(B) # eigh: faster, guarantees real output for symmetric
print("Eigenvalues (all ≥ 0 for PSD):", np.round(vals_b, 3))
print()
print("Orthogonality check V^T V (should be identity):")
print(np.round(vecs_b.T @ vecs_b, 10))
Section 10.4 — SVD and Low-Rank Approximation
The singular value decomposition (SVD) is the workhorse of modern computational linear algebra. Unlike the eigendecomposition, it applies to any $m \times n$ matrix — square or rectangular:
$$\boxed{\mathbf{A} = \mathbf{U}\,\mathbf{\Sigma}\,\mathbf{V}^T}$$
where $\mathbf{U} \in \mathbb{R}^{m \times m}$ (left singular vectors, orthonormal columns), $\mathbf{\Sigma} \in \mathbb{R}^{m \times n}$ (non-negative diagonal singular values $\sigma_1 \geq \sigma_2 \geq \ldots \geq 0$), and $\mathbf{V} \in \mathbb{R}^{n \times n}$ (right singular vectors, orthonormal columns). The singular values measure how much energy each direction carries: the first few usually account for 90–99 % of the total Frobenius-norm energy.
The Eckart–Young theorem (1936) states that the best rank-$r$ approximation of $\mathbf{A}$ in Frobenius norm is obtained by truncating to the top-$r$ singular triplets:
$$\boxed{\mathbf{A}_r = \sum_{i=1}^{r} \sigma_i\,\mathbf{u}_i\,\mathbf{v}_i^T \;=\; \mathbf{U}_{:,1:r}\,\mathbf{\Sigma}_{1:r,1:r}\,\mathbf{V}_{1:r,:}^T}$$
PCA via SVD: If $\tilde{\mathbf{X}}$ is the mean-centred data matrix ($n_{\text{samples}} \times n_{\text{features}}$), then the principal components are the right singular vectors of $\tilde{\mathbf{X}}$ (rows of $\mathbf{V}^T$), and the scores are $\mathbf{Z} = \tilde{\mathbf{X}}\mathbf{V}_{:,1:k}$. This is equivalent to diagonalising the sample covariance matrix but numerically more stable.
Applications beyond PCA: image/audio compression (Eckart–Young), collaborative filtering (Netflix prize uses matrix factorisation = approximate SVD), latent semantic indexing, and low-rank weight compression in large language models.
# === Section 10.4: SVD low-rank approximation of a random image ===
rng = np.random.default_rng(42)
img = rng.integers(0, 256, (64, 64)).astype(float)
U, s, Vt = np.linalg.svd(img, full_matrices=False)
ranks = [2, 5, 10, 20, 64]
fig, axes = plt.subplots(1, len(ranks), figsize=(14, 3))
for ax, r in zip(axes, ranks):
approx = U[:, :r] @ np.diag(s[:r]) @ Vt[:r, :]
rel_err = np.linalg.norm(img - approx, "fro") / np.linalg.norm(img, "fro")
ax.imshow(approx, cmap="gray", vmin=0, vmax=255)
ax.set_title(f"rank={r}\nerr={rel_err:.3f}")
ax.axis("off")
# Energy plot
cumulative_energy = np.cumsum(s**2) / np.sum(s**2)
print(f"Energy captured by top-5 singular values: {cumulative_energy[4]*100:.1f}%")
print(f"Energy captured by top-10 singular values: {cumulative_energy[9]*100:.1f}%")
plt.suptitle("SVD Low-Rank Approximations (64×64 random image)", y=1.02)
plt.tight_layout()
plt.show()
# === Section 10.4 (cont.): PCA via SVD on the UCI Wine dataset ===
data_path = pathlib.Path("../../data/wine_data.csv")
df_wine = pd.read_csv(data_path)
features = df_wine.select_dtypes("number").drop(columns=["target"], errors="ignore")
X_w = features.values.astype(float)
X_c = X_w - X_w.mean(axis=0) # mean-centre
_, _, Vt_w = np.linalg.svd(X_c, full_matrices=False)
Z = X_c @ Vt_w[:3].T # project onto top-3 PCs
label_col = "target" if "target" in df_wine.columns else df_wine.columns[0]
fig = px.scatter_3d(
x=Z[:, 0], y=Z[:, 1], z=Z[:, 2],
color=df_wine[label_col].astype(str),
title="PCA 3-D — UCI Wine Dataset (SVD method)",
labels={"x": "PC1", "y": "PC2", "z": "PC3"},
)
fig.show()
Section 10.5 — Solving Linear Systems
The fundamental problem of numerical linear algebra is: given $\mathbf{A}$ and $\mathbf{b}$, find $\mathbf{x}$ such that
$$\boxed{\mathbf{A}\,\mathbf{x} = \mathbf{b}}$$
This arises throughout ML. In OLS regression the normal equations give:
$$\boxed{\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}}$$
but in practice we never compute the matrix inverse explicitly — inverting $\mathbf{A}$ numerically doubles the floating-point error and runs in $O(n^3)$. Instead we use LU decomposition:
$$\boxed{\mathbf{A} = \mathbf{L}\,\mathbf{U}} \quad\Rightarrow\quad \mathbf{L}\,\mathbf{U}\,\mathbf{x} = \mathbf{b}$$
Solved in two triangular sweeps ($O(n^2)$ each after the $O(n^3)$ factorisation
is cached). scipy.linalg.solve does this automatically and is the recommended
API for all small-to-medium dense systems.
The condition number $\kappa(\mathbf{A}) = \sigma_{\max} / \sigma_{\min}$ (ratio of largest to smallest singular value) measures how sensitive the solution is to perturbations in $\mathbf{b}$ or $\mathbf{A}$. A rule of thumb: each decimal order of $\kappa$ costs roughly one digit of accuracy. If $\kappa \sim 10^{12}$ on 64-bit doubles (machine epsilon $\approx 10^{-16}$), the solution is unreliable. Ill-conditioned systems arise in polynomial regression, multicollinear features, and near-duplicate training samples.
# === Section 10.5: Solving Ax=b with scipy.linalg ===
A_sys = np.array([[ 3., 1., -1.],
[ 2., 4., 1.],
[-1., 2., 5.]])
b_sys = np.array([4., 1., 1.])
x_sol = linalg.solve(A_sys, b_sys)
residual = np.linalg.norm(A_sys @ x_sol - b_sys)
cond_num = np.linalg.cond(A_sys)
print(f"Solution x = {x_sol}")
print(f"Residual ‖Ax−b‖ = {residual:.2e}")
print(f"Condition number = {cond_num:.2f} ({'well' if cond_num < 100 else 'ill'}-conditioned)")
# Contrast: an ill-conditioned Hilbert matrix
n = 6
H = np.array([[1.0 / (i + j + 1) for j in range(n)] for i in range(n)])
print(f"\nHilbert matrix (n={n}) condition number = {np.linalg.cond(H):.2e} (ill-conditioned!)")
Section 10.6 — Summary & Quick Reference
Key Formulas¶
| Concept | Formula | Python |
|---|---|---|
| Dot product | $\mathbf{a}\cdot\mathbf{b} = \sum_i a_i b_i$ | a @ b |
| L2 norm | $\|\mathbf{a}\| = \sqrt{\mathbf{a}\cdot\mathbf{a}}$ | np.linalg.norm(a) |
| Cosine similarity | $\cos\theta = \dfrac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|}$ | (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b)) |
| Matrix multiply | $(AB)_{ij} = \sum_k A_{ik}B_{kj}$ | A @ B |
| Eigendecomposition | $\mathbf{A} = \mathbf{V}\mathbf{\Lambda}\mathbf{V}^{-1}$ | np.linalg.eig(A) |
| Symmetric eigen | $\mathbf{A} = \mathbf{V}\mathbf{\Lambda}\mathbf{V}^T$ | np.linalg.eigh(A) |
| SVD | $\mathbf{A} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T$ | np.linalg.svd(A) |
| Low-rank approx | $\mathbf{A}_r = \mathbf{U}_{:r}\mathbf{\Sigma}_{:r}\mathbf{V}^T_{:r}$ | slice U, s, Vt |
| Linear solve | $\mathbf{A}\mathbf{x} = \mathbf{b}$ | scipy.linalg.solve(A, b) |
| OLS normal eqs | $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$ | np.linalg.lstsq(X, y) |
| Condition number | $\kappa(\mathbf{A}) = \sigma_{\max}/\sigma_{\min}$ | np.linalg.cond(A) |
ML Connections¶
| Concept | Where It Appears |
|---|---|
| Dot product | Attention scores, kernel evaluations, recommendation similarity |
| Matrix multiply | Every nn.Linear layer: $\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}$ |
| Eigendecomposition | PCA, spectral clustering, graph neural networks |
| SVD | Recommender systems, weight compression, latent semantic analysis |
| Linear solve | OLS, Kalman filters, Gaussian process posterior |
| Condition number | Gradient vanishing/exploding, preconditioning, batch normalisation |
What's Next?¶
$$\boxed{\text{Ch 10: Linear Algebra}} \;\to\; \boxed{\text{Ch 11: Vector Calculus}} \;\to\; \boxed{\text{Ch 12: Scientific Python}} \;\to\; \boxed{\text{Ch 13: Probability \& Info Theory}} \;\to\; \boxed{\text{Ch 14: Neural Networks}}$$
Chapter 11 adds the calculus layer: partial derivatives, the Jacobian, the Hessian, the chain rule, and gradient descent — the optimisation engine that trains every neural network. The matrix notation you just learned is exactly the language in which backpropagation is written.
References¶
Strang, G. (2023). Introduction to Linear Algebra (6th ed.). Wellesley–Cambridge Press. The standard MIT undergraduate reference; Chapters 1–7 cover all material in this chapter. https://math.mit.edu/~gs/linearalgebra/
Trefethen, L. N., & Bau, D. (1997). Numerical Linear Algebra. SIAM. Graduate-level treatment of SVD, QR factorisation, and numerical stability; essential reading for anyone implementing solvers.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 2 (Linear Algebra) provides the ML-focused perspective on all topics in this chapter. https://www.deeplearningbook.org/
Bishop, C. M., & Bishop, H. (2024). Deep Learning: Foundations and Concepts. Springer. Chapter 1 reviews linear algebra in the context of modern deep learning. https://www.bishopbook.com/
Eckart, C., & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1(3), 211–218. Original proof of the best low-rank approximation theorem (the theoretical basis of SVD truncation).
Wikipedia. Singular value decomposition. https://en.wikipedia.org/wiki/Singular_value_decomposition
Wikipedia. Eigendecomposition of a matrix. https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix
NumPy Development Team. NumPy Linear Algebra Reference. https://numpy.org/doc/stable/reference/routines.linalg.html