Statistical Basis of AI/ML — Vector Calculus and Optimization
Chapter 11 — Gradients, the Chain Rule, and the Engine of Learning
Why This Chapter?
In Chapter 6 you fitted a linear model by minimising the sum of squared
residuals — a convex quadratic that has a closed-form solution. In Chapters
7 and 8 you trained classifiers and ran K-Means by calling .fit(). In
Chapter 9 you watched XGBoost converge iteration by iteration, each round
fitting a tree to the gradient of the previous loss. In all of these cases,
an optimiser was working quietly underneath the API, searching for parameter
values that make a loss function as small as possible. This chapter makes that
machinery explicit.
The central tool is the gradient — a vector of partial derivatives that points in the direction of steepest ascent of a function. Gradient descent (and its many descendants: momentum, RMSProp, Adam) follows the negative gradient to find a local minimum. The gradient of a scalar with respect to a vector parameter is a vector; the gradient of a vector-valued function with respect to a vector is a matrix called the Jacobian; the second-order generalisation is the Hessian, which encodes local curvature.
For deep networks with millions of parameters, computing all partial derivatives by hand would be impossible. The chain rule solves this: if $z$ is a function of $y$, which is a function of $x$, then $dz/dx = (dz/dy)(dy/dx)$. In vector form this becomes the product of Jacobians, and backpropagation is simply a systematic application of the chain rule traversing the computation graph in reverse. Understanding the chain rule at the scalar level is the essential prerequisite for understanding backprop at the matrix level.
This chapter develops all of this from first principles: symbolic differentiation
with SymPy, finite-difference verification, manual gradient-descent implementation,
and then the production-quality quasi-Newton optimizer (scipy.optimize.minimize
with method="BFGS") that shows how second-order curvature information
dramatically accelerates convergence on non-trivial loss surfaces.
$$\boxed{\text{Partial Derivatives}} \;\xrightarrow{\text{stack}}\; \boxed{\text{Gradient}} \;\xrightarrow{\text{compose}}\; \boxed{\text{Chain Rule}} \;\xrightarrow{\text{iterate}}\; \boxed{\text{Gradient Descent}}$$
What You Will Learn
| $\S$ | Topic | Description |
|---|---|---|
| 11.0 | Setup | Imports & configuration |
| 11.1 | Symbolic Gradient and Gradient Field | SymPy differentiation; contour + quiver visualisation |
| 11.2 | Jacobian and Hessian | Vector-to-vector derivatives; second-order curvature |
| 11.3 | Chain Rule — Analytic vs Finite Difference | Verification of analytic gradients; finite-difference check |
| 11.4 | Manual Gradient Descent | From-scratch implementation; convergence plot; 2-D trajectory |
| 11.5 | scipy.optimize with Explicit Gradient |
BFGS on the Rosenbrock function; comparing methods |
| 11.6 | Summary & Quick Reference | Formulas, ML connections, optimizer cheat sheet |
| 11.7 | References | Authoritative sources |
Section 11.0 — Setup: Imports & Configuration
This chapter uses SymPy for symbolic differentiation and SciPy for production-grade numerical optimisation. All other libraries are the standard toolkit established in Chapter 2.
| Library | Role in This Chapter |
|---|---|
| NumPy | Vectorised arithmetic, finite-difference gradient checks |
| SymPy | Exact symbolic differentiation — gradients, Jacobians, Hessians |
| Matplotlib | Contour plots, quiver (vector field) plots, convergence curves |
SciPy optimize |
minimize with analytic Jacobian; BFGS, L-BFGS-B methods |
| Seaborn | Plot aesthetics |
# === Section 11.0: Imports & configuration ===
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.optimize import minimize
import sympy as sp
rng = np.random.default_rng(42)
sns.set_theme(style="whitegrid", palette="muted")
print("NumPy", np.__version__, "| SymPy", sp.__version__, "| SciPy minimize ready")
Section 11.1 — Symbolic Gradient and Gradient Field
The gradient of a scalar function $f : \mathbb{R}^n \to \mathbb{R}$ is the vector of its partial derivatives:
$$\boxed{\nabla f(\mathbf{x}) = \begin{pmatrix}\dfrac{\partial f}{\partial x_1} \\ \vdots \\ \dfrac{\partial f}{\partial x_n}\end{pmatrix}}$$
It points in the direction of steepest ascent of $f$ at $\mathbf{x}$. The negative gradient therefore points toward the nearest local minimum — which is exactly the direction a gradient-descent optimiser takes. The magnitude $\|\nabla f(\mathbf{x})\|$ tells you how steeply the function is rising (or, in descent, falling).
For the example $f(x,y) = x^2 + 2y^2 - xy$ we have:
$$\nabla f = \begin{pmatrix}2x - y \\ 4y - x\end{pmatrix}$$
SymPy computes this symbolically — exact rational arithmetic, no floating-point
error. We then convert to numeric functions via lambdify to draw the gradient
field. The quiver plot below shows the negative gradient (descent direction)
overlaid on the contour map of $f$ — you can visually trace how a particle
starting anywhere would roll toward the minimum at the origin.
# === Section 11.1: Symbolic gradient with SymPy ===
x, y = sp.symbols("x y", real=True)
f_sym = x**2 + 2*y**2 - x*y
grad_f = [sp.diff(f_sym, v) for v in (x, y)]
print("f(x,y) =", f_sym)
print("∂f/∂x =", grad_f[0])
print("∂f/∂y =", grad_f[1])
# Critical point: set gradient to zero
critical = sp.solve(grad_f, [x, y])
print("Critical point:", critical)
# === Section 11.1 (cont.): Gradient field visualisation ===
f_np = sp.lambdify((x, y), f_sym, "numpy")
gx_np = sp.lambdify((x, y), grad_f[0], "numpy")
gy_np = sp.lambdify((x, y), grad_f[1], "numpy")
xx, yy = np.meshgrid(np.linspace(-3, 3, 25), np.linspace(-3, 3, 25))
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
cs = axes[0].contourf(xx, yy, f_np(xx, yy), levels=20, cmap="viridis")
plt.colorbar(cs, ax=axes[0])
axes[0].set_title(r"$f(x,y) = x^2 + 2y^2 - xy$ (contour)")
axes[0].set_xlabel("x"); axes[0].set_ylabel("y")
axes[1].quiver(xx, yy, -gx_np(xx, yy), -gy_np(xx, yy),
alpha=0.75, color="steelblue")
axes[1].set_title("Negative gradient field (descent directions)")
axes[1].set_xlabel("x"); axes[1].set_ylabel("y")
plt.tight_layout()
plt.show()
Section 11.2 — Jacobian and Hessian
When the output of a function is also a vector, the generalisation of the gradient is the Jacobian matrix $\mathbf{J}$:
$$\boxed{J_{ij} = \frac{\partial F_i}{\partial x_j} \quad\Rightarrow\quad \mathbf{J}_{\mathbf{F}}(\mathbf{x}) \in \mathbb{R}^{m \times n}}$$
for $\mathbf{F} : \mathbb{R}^n \to \mathbb{R}^m$. In backpropagation, every layer's backward pass multiplies by its Jacobian transposed: $\delta\mathbf{x} = \mathbf{J}^T\,\delta\mathbf{F}$.
The Hessian of a scalar $g : \mathbb{R}^n \to \mathbb{R}$ is the matrix of second partial derivatives:
$$\boxed{H_{ij} = \frac{\partial^2 g}{\partial x_i \partial x_j} \quad\Rightarrow\quad \mathbf{H}_g(\mathbf{x}) \in \mathbb{R}^{n \times n}}$$
The Hessian encodes curvature: positive definiteness means the current point is a local minimum; negative definiteness means a local maximum; indefinite means a saddle point. Second-order optimisers (Newton's method, BFGS, natural gradient) use Hessian information to adapt the step size per direction, converging much faster than first-order gradient descent on ill-conditioned loss surfaces.
# === Section 11.2: Jacobian and Hessian with SymPy ===
x1, x2 = sp.symbols("x1 x2", real=True)
F = sp.Matrix([x1**2 + x2,
sp.sin(x1) * x2**2])
J = F.jacobian([x1, x2])
print("F(x1,x2) =")
sp.pprint(F)
print("\nJacobian J =")
sp.pprint(J)
print("\nJ evaluated at (π/2, 1):")
sp.pprint(J.subs({x1: sp.pi/2, x2: 1}))
# Hessian of a scalar function
g = x1**2 + 3*x1*x2 + x2**3
H = sp.hessian(g, [x1, x2])
print("\n\ng(x1,x2) =", g)
print("Hessian H =")
sp.pprint(H)
Section 11.3 — Chain Rule: Analytic vs Finite Difference
The chain rule is the cornerstone of automatic differentiation. For a composition $h(x) = f(g(x))$:
$$\boxed{\frac{dh}{dx} = f'(g(x)) \cdot g'(x)}$$
In the multi-variable case, with $\mathbf{y} = \mathbf{f}(\mathbf{g}(\mathbf{x}))$:
$$\boxed{\frac{\partial \mathbf{y}}{\partial \mathbf{x}} = \frac{\partial \mathbf{f}}{\partial \mathbf{g}} \cdot \frac{\partial \mathbf{g}}{\partial \mathbf{x}} = \mathbf{J}_{\mathbf{f}} \cdot \mathbf{J}_{\mathbf{g}}}$$
Backpropagation is the chain rule applied recursively from the loss backward through every layer. Before trusting an analytic gradient, practitioners verify it against the finite-difference approximation (central difference):
$$\boxed{\frac{df}{dx}\bigg|_{x=x_0} \approx \frac{f(x_0 + h) - f(x_0 - h)}{2h}, \quad h \approx 10^{-5}}$$
The central difference is $O(h^2)$ accurate — much better than the one-sided $O(h)$ forward difference. A residual below $10^{-6}$ between analytic and finite-difference gradients is strong evidence the analytic expression is correct. This gradient check is standard practice when implementing custom backward passes in PyTorch or JAX.
# === Section 11.3: Chain rule — analytic gradient vs finite difference ===
# y = exp(sin(x²))
# dy/dx = exp(sin(x²)) · cos(x²) · 2x (chain rule applied twice)
def y_func(x):
return np.exp(np.sin(x**2))
def dy_analytic(x):
return np.exp(np.sin(x**2)) * np.cos(x**2) * 2.0 * x
xs = np.linspace(0.1, 2.0, 300)
h = 1e-5
dy_fd = (y_func(xs + h) - y_func(xs - h)) / (2.0 * h) # central difference
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].plot(xs, dy_analytic(xs), lw=2, label="Analytic")
axes[0].plot(xs, dy_fd, "--", lw=1.5, alpha=.8, label=f"Finite diff h={h:.0e}")
axes[0].set_title(r"$dy/dx$ for $y = e^{\sin(x^2)}$")
axes[0].legend(); axes[0].set_xlabel("x")
err = np.abs(dy_analytic(xs) - dy_fd)
axes[1].semilogy(xs, err, color="tomato")
axes[1].set_title("Absolute error |analytic − FD|")
axes[1].set_xlabel("x"); axes[1].set_ylabel("error (log scale)")
plt.tight_layout(); plt.show()
print(f"Max |analytic − FD| = {err.max():.2e} (should be < 1e-7)")
Section 11.4 — Manual Gradient Descent
Gradient descent updates the parameter vector $\mathbf{w}$ by stepping opposite the gradient:
$$\boxed{\mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \eta\,\nabla_{\mathbf{w}} \mathcal{L}(\mathbf{w}^{(t)})}$$
where $\eta > 0$ is the learning rate (step size). Too large: the iterates overshoot and may diverge. Too small: convergence is unnecessarily slow. Choosing $\eta$ well is one of the core skills of deep learning practitioners.
For the toy loss $\mathcal{L}(w_1, w_2) = (w_1 - 2)^2 + 0.5(w_2 + 3)^2$, the analytic gradient is:
$$\nabla \mathcal{L} = \begin{pmatrix}2(w_1 - 2) \\ w_2 + 3\end{pmatrix}$$
The minimum is at $(w_1^*, w_2^*) = (2, -3)$. Note the anisotropy: the $w_1$ bowl is twice as curved as the $w_2$ bowl (Hessian eigenvalues are 2 and 1). A single learning rate must therefore be a compromise — this motivates adaptive methods like Adam that maintain a per-parameter effective step size.
# === Section 11.4: Manual gradient descent ===
def loss(w): return (w[0] - 2)**2 + 0.5*(w[1] + 3)**2
def grad_l(w): return np.array([2.0*(w[0] - 2), w[1] + 3])
w = np.array([5.0, 5.0])
lr = 0.1
history = [w.copy()]
for _ in range(80):
w = w - lr * grad_l(w)
history.append(w.copy())
history = np.array(history)
print(f"Final w = {w} (target: [2, -3])")
print(f"Final L = {loss(w):.2e}")
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
# Convergence
axes[0].plot(history[:, 0], label="w₁ (target=2)")
axes[0].plot(history[:, 1], label="w₂ (target=−3)")
axes[0].axhline( 2, ls="--", c="steelblue", alpha=.4)
axes[0].axhline(-3, ls="--", c="tomato", alpha=.4)
axes[0].legend(); axes[0].set_title(f"Parameter convergence (η={lr})")
axes[0].set_xlabel("iteration"); axes[0].set_ylabel("value")
# 2-D trajectory
ww1, ww2 = np.meshgrid(np.linspace(-1, 6, 100), np.linspace(-6, 6, 100))
L_grid = (ww1 - 2)**2 + 0.5*(ww2 + 3)**2
cs = axes[1].contourf(ww1, ww2, L_grid, levels=25, cmap="plasma", alpha=.7)
plt.colorbar(cs, ax=axes[1])
axes[1].plot(history[:, 0], history[:, 1], "o-", ms=3, color="white", lw=1, label="trajectory")
axes[1].scatter([2], [-3], marker="*", s=250, color="gold", zorder=5, label="minimum")
axes[1].legend(); axes[1].set_title("Gradient-descent trajectory in loss landscape")
axes[1].set_xlabel("w₁"); axes[1].set_ylabel("w₂")
plt.tight_layout(); plt.show()
Section 11.5 — <code>scipy.optimize</code> with Explicit Gradient
Real optimisation problems are rarely convex quadratics. The Rosenbrock function (also called the "banana function") is a classic non-convex test case:
$$\boxed{f(w_1, w_2) = (1 - w_1)^2 + 100(w_2 - w_1^2)^2}$$
It has a narrow, curved valley and a unique global minimum at $(1, 1)$ that is hard to reach for first-order methods with fixed step sizes. The analytic gradient is:
$$\nabla f = \begin{pmatrix}-2(1-w_1) - 400 w_1(w_2 - w_1^2) \\ 200(w_2 - w_1^2)\end{pmatrix}$$
scipy.optimize.minimize with method="BFGS" (Broyden–Fletcher–Goldfarb–
Shanno) is a quasi-Newton method that builds an approximation to the
Hessian using successive gradient evaluations. Unlike vanilla gradient
descent, BFGS adapts its step to the local curvature and converges
superlinearly near the minimum. Providing the analytic Jacobian via jac=
makes each step more accurate and roughly halves the number of function
evaluations compared to finite-difference gradients.
# === Section 11.5: BFGS on the Rosenbrock function ===
def rosenbrock(w):
return (1.0 - w[0])**2 + 100.0*(w[1] - w[0]**2)**2
def rosenbrock_grad(w):
dw0 = -2.0*(1.0 - w[0]) - 400.0*w[0]*(w[1] - w[0]**2)
dw1 = 200.0*(w[1] - w[0]**2)
return np.array([dw0, dw1])
w0 = np.array([-1.5, 2.0])
res = minimize(rosenbrock, w0, jac=rosenbrock_grad, method="BFGS",
options={"gtol": 1e-8})
print(f"Converged : {res.success}")
print(f"Minimum at : {res.x} (target: [1, 1])")
print(f"f(w*) : {res.fun:.2e}")
print(f"Iterations : {res.nit}")
print(f"Grad evals : {res.njev}")
# Visual: loss landscape + final solution
ww1, ww2 = np.meshgrid(np.linspace(-2, 2, 300), np.linspace(-1, 3, 300))
Z = rosenbrock([ww1, ww2])
fig, ax = plt.subplots(figsize=(7, 5))
cs = ax.contourf(ww1, ww2, np.log1p(Z), levels=40, cmap="magma")
plt.colorbar(cs, ax=ax, label="log(1 + f)")
ax.scatter(*res.x, marker="*", s=300, color="gold", zorder=5, label=f"BFGS min {res.x}")
ax.scatter(*w0, marker="o", s=100, color="cyan", zorder=5, label=f"start {w0}")
ax.set_title("Rosenbrock landscape (log scale)"); ax.legend()
ax.set_xlabel("w₁"); ax.set_ylabel("w₂")
plt.tight_layout(); plt.show()
Section 11.6 — Summary & Quick Reference
Key Formulas¶
| Concept | Formula | Python |
|---|---|---|
| Gradient | $\nabla f = [\partial f/\partial x_1, \ldots, \partial f/\partial x_n]^T$ | sp.diff(f, x) / jac= |
| Jacobian | $J_{ij} = \partial F_i / \partial x_j$ | F.jacobian([x1, x2]) |
| Hessian | $H_{ij} = \partial^2 g / \partial x_i \partial x_j$ | sp.hessian(g, vars) |
| Chain rule | $dz/dx = (dz/dy)(dy/dx)$ | automatic via autograd/SymPy |
| Central FD | $(f(x+h)-f(x-h))/2h$ | manual or scipy.optimize.approx_fprime |
| GD update | $\mathbf{w} \leftarrow \mathbf{w} - \eta\nabla\mathcal{L}$ | w -= lr * grad(w) |
| BFGS | quasi-Newton, superlinear convergence | minimize(f, w0, jac=g, method="BFGS") |
ML Connections¶
| Concept | Where It Appears |
|---|---|
| Gradient | Gradient descent, AdaGrad, RMSProp, Adam — all follow $-\nabla\mathcal{L}$ |
| Jacobian | Backward pass of any layer: $\delta\mathbf{x} = \mathbf{J}^T\,\delta\mathbf{y}$ |
| Hessian | Newton's method, natural gradient, Fisher information matrix |
| Chain rule | Backpropagation — the chain rule applied through the computation graph |
| FD gradient check | Unit-testing custom backward implementations |
| BFGS / L-BFGS | Fine-tuning, hyperparameter optimisation, second-order training |
Optimizer Comparison¶
| Method | Info used | Convergence | Memory | Use when |
|---|---|---|---|---|
| Gradient Descent | $\nabla\mathcal{L}$ | Linear | $O(n)$ | Simple demos, convex problems |
| SGD + momentum | $\nabla\mathcal{L}$ mini-batch | Sub-linear | $O(n)$ | Large neural nets |
| Adam | adaptive per-param | Fast in practice | $O(n)$ | Default for deep learning |
| BFGS | $\nabla\mathcal{L}$ + approx $\mathbf{H}^{-1}$ | Superlinear | $O(n^2)$ | Small parameter spaces |
| L-BFGS | limited-memory BFGS | Superlinear | $O(mn)$ | Medium problems, fine-tuning |
What's Next?¶
$$\boxed{\text{Ch 11: Calculus}} \;\to\; \boxed{\text{Ch 12: Scientific Python}} \;\to\; \boxed{\text{Ch 13: Information Theory}} \;\to\; \boxed{\text{Ch 14: Neural Networks}}$$
Chapter 12 shifts from mathematics to engineering: how do you make the computations you just learned run 100× faster? Vectorisation, broadcasting, and advanced NumPy patterns turn the calculus of this chapter into production-ready code.
References¶
Spivak, M. (1965). Calculus on Manifolds. W. A. Benjamin. The definitive rigorous treatment of multivariable calculus and the generalised chain rule.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 4 (Numerical Computation) and Chapter 6 (Deep Feedforward Networks — backpropagation derivation). https://www.deeplearningbook.org/
Nocedal, J., & Wright, S. J. (2006). Numerical Optimization (2nd ed.). Springer. The graduate reference for gradient descent, quasi-Newton methods (BFGS), and line-search convergence theory.
Bishop, C. M., & Bishop, H. (2024). Deep Learning: Foundations and Concepts. Springer. Chapter 7 covers gradient-based optimisation in the modern deep-learning context. https://www.bishopbook.com/
Wikipedia. Gradient. https://en.wikipedia.org/wiki/Gradient
Wikipedia. Chain rule. https://en.wikipedia.org/wiki/Chain_rule
Wikipedia. Backpropagation. https://en.wikipedia.org/wiki/Backpropagation
Wikipedia. Broyden–Fletcher–Goldfarb–Shanno algorithm. https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm
SciPy Development Team. scipy.optimize.minimize documentation. https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html
SymPy Development Team. SymPy calculus reference. https://docs.sympy.org/latest/modules/calculus/