Home / About Us / Volume I Contents / Ch 11: Vector Calculus and Optimization

Statistical Basis of AI/ML — Vector Calculus and Optimization

Chapter 11 — Gradients, the Chain Rule, and the Engine of Learning


Why This Chapter?

In Chapter 6 you fitted a linear model by minimising the sum of squared residuals — a convex quadratic that has a closed-form solution. In Chapters 7 and 8 you trained classifiers and ran K-Means by calling .fit(). In Chapter 9 you watched XGBoost converge iteration by iteration, each round fitting a tree to the gradient of the previous loss. In all of these cases, an optimiser was working quietly underneath the API, searching for parameter values that make a loss function as small as possible. This chapter makes that machinery explicit.

The central tool is the gradient — a vector of partial derivatives that points in the direction of steepest ascent of a function. Gradient descent (and its many descendants: momentum, RMSProp, Adam) follows the negative gradient to find a local minimum. The gradient of a scalar with respect to a vector parameter is a vector; the gradient of a vector-valued function with respect to a vector is a matrix called the Jacobian; the second-order generalisation is the Hessian, which encodes local curvature.

For deep networks with millions of parameters, computing all partial derivatives by hand would be impossible. The chain rule solves this: if $z$ is a function of $y$, which is a function of $x$, then $dz/dx = (dz/dy)(dy/dx)$. In vector form this becomes the product of Jacobians, and backpropagation is simply a systematic application of the chain rule traversing the computation graph in reverse. Understanding the chain rule at the scalar level is the essential prerequisite for understanding backprop at the matrix level.

This chapter develops all of this from first principles: symbolic differentiation with SymPy, finite-difference verification, manual gradient-descent implementation, and then the production-quality quasi-Newton optimizer (scipy.optimize.minimize with method="BFGS") that shows how second-order curvature information dramatically accelerates convergence on non-trivial loss surfaces.

$$\boxed{\text{Partial Derivatives}} \;\xrightarrow{\text{stack}}\; \boxed{\text{Gradient}} \;\xrightarrow{\text{compose}}\; \boxed{\text{Chain Rule}} \;\xrightarrow{\text{iterate}}\; \boxed{\text{Gradient Descent}}$$

What You Will Learn

$\S$ Topic Description
11.0 Setup Imports & configuration
11.1 Symbolic Gradient and Gradient Field SymPy differentiation; contour + quiver visualisation
11.2 Jacobian and Hessian Vector-to-vector derivatives; second-order curvature
11.3 Chain Rule — Analytic vs Finite Difference Verification of analytic gradients; finite-difference check
11.4 Manual Gradient Descent From-scratch implementation; convergence plot; 2-D trajectory
11.5 scipy.optimize with Explicit Gradient BFGS on the Rosenbrock function; comparing methods
11.6 Summary & Quick Reference Formulas, ML connections, optimizer cheat sheet
11.7 References Authoritative sources

Section 11.0 — Setup: Imports & Configuration

This chapter uses SymPy for symbolic differentiation and SciPy for production-grade numerical optimisation. All other libraries are the standard toolkit established in Chapter 2.

Library Role in This Chapter
NumPy Vectorised arithmetic, finite-difference gradient checks
SymPy Exact symbolic differentiation — gradients, Jacobians, Hessians
Matplotlib Contour plots, quiver (vector field) plots, convergence curves
SciPy optimize minimize with analytic Jacobian; BFGS, L-BFGS-B methods
Seaborn Plot aesthetics
In [1]:
# === Section 11.0: Imports & configuration ===
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.optimize import minimize
import sympy as sp

rng = np.random.default_rng(42)
sns.set_theme(style="whitegrid", palette="muted")
print("NumPy", np.__version__, "| SymPy", sp.__version__, "| SciPy minimize ready")
NumPy 2.3.5 | SymPy 1.14.0 | SciPy minimize ready

Section 11.1 — Symbolic Gradient and Gradient Field

The gradient of a scalar function $f : \mathbb{R}^n \to \mathbb{R}$ is the vector of its partial derivatives:

$$\boxed{\nabla f(\mathbf{x}) = \begin{pmatrix}\dfrac{\partial f}{\partial x_1} \\ \vdots \\ \dfrac{\partial f}{\partial x_n}\end{pmatrix}}$$

It points in the direction of steepest ascent of $f$ at $\mathbf{x}$. The negative gradient therefore points toward the nearest local minimum — which is exactly the direction a gradient-descent optimiser takes. The magnitude $\|\nabla f(\mathbf{x})\|$ tells you how steeply the function is rising (or, in descent, falling).

For the example $f(x,y) = x^2 + 2y^2 - xy$ we have:

$$\nabla f = \begin{pmatrix}2x - y \\ 4y - x\end{pmatrix}$$

SymPy computes this symbolically — exact rational arithmetic, no floating-point error. We then convert to numeric functions via lambdify to draw the gradient field. The quiver plot below shows the negative gradient (descent direction) overlaid on the contour map of $f$ — you can visually trace how a particle starting anywhere would roll toward the minimum at the origin.

In [2]:
# === Section 11.1: Symbolic gradient with SymPy ===
x, y = sp.symbols("x y", real=True)
f_sym = x**2 + 2*y**2 - x*y

grad_f = [sp.diff(f_sym, v) for v in (x, y)]
print("f(x,y)  =", f_sym)
print("∂f/∂x   =", grad_f[0])
print("∂f/∂y   =", grad_f[1])

# Critical point: set gradient to zero
critical = sp.solve(grad_f, [x, y])
print("Critical point:", critical)
f(x,y)  = x**2 - x*y + 2*y**2
∂f/∂x   = 2*x - y
∂f/∂y   = -x + 4*y
Critical point: {x: 0, y: 0}
In [3]:
# === Section 11.1 (cont.): Gradient field visualisation ===
f_np  = sp.lambdify((x, y), f_sym,     "numpy")
gx_np = sp.lambdify((x, y), grad_f[0], "numpy")
gy_np = sp.lambdify((x, y), grad_f[1], "numpy")

xx, yy = np.meshgrid(np.linspace(-3, 3, 25), np.linspace(-3, 3, 25))

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

cs = axes[0].contourf(xx, yy, f_np(xx, yy), levels=20, cmap="viridis")
plt.colorbar(cs, ax=axes[0])
axes[0].set_title(r"$f(x,y) = x^2 + 2y^2 - xy$  (contour)")
axes[0].set_xlabel("x"); axes[0].set_ylabel("y")

axes[1].quiver(xx, yy, -gx_np(xx, yy), -gy_np(xx, yy),
               alpha=0.75, color="steelblue")
axes[1].set_title("Negative gradient field (descent directions)")
axes[1].set_xlabel("x"); axes[1].set_ylabel("y")

plt.tight_layout()
plt.show()
No description has been provided for this image

Section 11.2 — Jacobian and Hessian

When the output of a function is also a vector, the generalisation of the gradient is the Jacobian matrix $\mathbf{J}$:

$$\boxed{J_{ij} = \frac{\partial F_i}{\partial x_j} \quad\Rightarrow\quad \mathbf{J}_{\mathbf{F}}(\mathbf{x}) \in \mathbb{R}^{m \times n}}$$

for $\mathbf{F} : \mathbb{R}^n \to \mathbb{R}^m$. In backpropagation, every layer's backward pass multiplies by its Jacobian transposed: $\delta\mathbf{x} = \mathbf{J}^T\,\delta\mathbf{F}$.

The Hessian of a scalar $g : \mathbb{R}^n \to \mathbb{R}$ is the matrix of second partial derivatives:

$$\boxed{H_{ij} = \frac{\partial^2 g}{\partial x_i \partial x_j} \quad\Rightarrow\quad \mathbf{H}_g(\mathbf{x}) \in \mathbb{R}^{n \times n}}$$

The Hessian encodes curvature: positive definiteness means the current point is a local minimum; negative definiteness means a local maximum; indefinite means a saddle point. Second-order optimisers (Newton's method, BFGS, natural gradient) use Hessian information to adapt the step size per direction, converging much faster than first-order gradient descent on ill-conditioned loss surfaces.

In [4]:
# === Section 11.2: Jacobian and Hessian with SymPy ===
x1, x2 = sp.symbols("x1 x2", real=True)

F = sp.Matrix([x1**2 + x2,
               sp.sin(x1) * x2**2])

J = F.jacobian([x1, x2])
print("F(x1,x2) =")
sp.pprint(F)
print("\nJacobian J =")
sp.pprint(J)
print("\nJ evaluated at (π/2, 1):")
sp.pprint(J.subs({x1: sp.pi/2, x2: 1}))

# Hessian of a scalar function
g = x1**2 + 3*x1*x2 + x2**3
H = sp.hessian(g, [x1, x2])
print("\n\ng(x1,x2) =", g)
print("Hessian H =")
sp.pprint(H)
F(x1,x2) =
⎡   2       ⎤
⎢ x₁  + x₂  ⎥
⎢           ⎥
⎢  2        ⎥
⎣x₂ ⋅sin(x₁)⎦

Jacobian J =
⎡   2⋅x₁           1      ⎤
⎢                         ⎥
⎢  2                      ⎥
⎣x₂ ⋅cos(x₁)  2⋅x₂⋅sin(x₁)⎦

J evaluated at (π/2, 1):
⎡π  1⎤
⎢    ⎥
⎣0  2⎦


g(x1,x2) = x1**2 + 3*x1*x2 + x2**3
Hessian H =
⎡2   3  ⎤
⎢       ⎥
⎣3  6⋅x₂⎦

Section 11.3 — Chain Rule: Analytic vs Finite Difference

The chain rule is the cornerstone of automatic differentiation. For a composition $h(x) = f(g(x))$:

$$\boxed{\frac{dh}{dx} = f'(g(x)) \cdot g'(x)}$$

In the multi-variable case, with $\mathbf{y} = \mathbf{f}(\mathbf{g}(\mathbf{x}))$:

$$\boxed{\frac{\partial \mathbf{y}}{\partial \mathbf{x}} = \frac{\partial \mathbf{f}}{\partial \mathbf{g}} \cdot \frac{\partial \mathbf{g}}{\partial \mathbf{x}} = \mathbf{J}_{\mathbf{f}} \cdot \mathbf{J}_{\mathbf{g}}}$$

Backpropagation is the chain rule applied recursively from the loss backward through every layer. Before trusting an analytic gradient, practitioners verify it against the finite-difference approximation (central difference):

$$\boxed{\frac{df}{dx}\bigg|_{x=x_0} \approx \frac{f(x_0 + h) - f(x_0 - h)}{2h}, \quad h \approx 10^{-5}}$$

The central difference is $O(h^2)$ accurate — much better than the one-sided $O(h)$ forward difference. A residual below $10^{-6}$ between analytic and finite-difference gradients is strong evidence the analytic expression is correct. This gradient check is standard practice when implementing custom backward passes in PyTorch or JAX.

In [5]:
# === Section 11.3: Chain rule — analytic gradient vs finite difference ===
# y = exp(sin(x²))
# dy/dx = exp(sin(x²)) · cos(x²) · 2x   (chain rule applied twice)

def y_func(x):
    return np.exp(np.sin(x**2))

def dy_analytic(x):
    return np.exp(np.sin(x**2)) * np.cos(x**2) * 2.0 * x

xs = np.linspace(0.1, 2.0, 300)
h  = 1e-5
dy_fd = (y_func(xs + h) - y_func(xs - h)) / (2.0 * h)   # central difference

fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].plot(xs, dy_analytic(xs), lw=2, label="Analytic")
axes[0].plot(xs, dy_fd, "--", lw=1.5, alpha=.8, label=f"Finite diff  h={h:.0e}")
axes[0].set_title(r"$dy/dx$ for $y = e^{\sin(x^2)}$")
axes[0].legend(); axes[0].set_xlabel("x")

err = np.abs(dy_analytic(xs) - dy_fd)
axes[1].semilogy(xs, err, color="tomato")
axes[1].set_title("Absolute error |analytic − FD|")
axes[1].set_xlabel("x"); axes[1].set_ylabel("error (log scale)")

plt.tight_layout(); plt.show()
print(f"Max |analytic − FD| = {err.max():.2e}  (should be < 1e-7)")
No description has been provided for this image
Max |analytic − FD| = 1.69e-09  (should be < 1e-7)

Section 11.4 — Manual Gradient Descent

Gradient descent updates the parameter vector $\mathbf{w}$ by stepping opposite the gradient:

$$\boxed{\mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \eta\,\nabla_{\mathbf{w}} \mathcal{L}(\mathbf{w}^{(t)})}$$

where $\eta > 0$ is the learning rate (step size). Too large: the iterates overshoot and may diverge. Too small: convergence is unnecessarily slow. Choosing $\eta$ well is one of the core skills of deep learning practitioners.

For the toy loss $\mathcal{L}(w_1, w_2) = (w_1 - 2)^2 + 0.5(w_2 + 3)^2$, the analytic gradient is:

$$\nabla \mathcal{L} = \begin{pmatrix}2(w_1 - 2) \\ w_2 + 3\end{pmatrix}$$

The minimum is at $(w_1^*, w_2^*) = (2, -3)$. Note the anisotropy: the $w_1$ bowl is twice as curved as the $w_2$ bowl (Hessian eigenvalues are 2 and 1). A single learning rate must therefore be a compromise — this motivates adaptive methods like Adam that maintain a per-parameter effective step size.

In [6]:
# === Section 11.4: Manual gradient descent ===
def loss(w):      return (w[0] - 2)**2 + 0.5*(w[1] + 3)**2
def grad_l(w):    return np.array([2.0*(w[0] - 2), w[1] + 3])

w       = np.array([5.0, 5.0])
lr      = 0.1
history = [w.copy()]

for _ in range(80):
    w = w - lr * grad_l(w)
    history.append(w.copy())

history = np.array(history)
print(f"Final w  = {w}  (target: [2, -3])")
print(f"Final L  = {loss(w):.2e}")

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Convergence
axes[0].plot(history[:, 0], label="w₁ (target=2)")
axes[0].plot(history[:, 1], label="w₂ (target=−3)")
axes[0].axhline( 2, ls="--", c="steelblue", alpha=.4)
axes[0].axhline(-3, ls="--", c="tomato",    alpha=.4)
axes[0].legend(); axes[0].set_title(f"Parameter convergence  (η={lr})")
axes[0].set_xlabel("iteration"); axes[0].set_ylabel("value")

# 2-D trajectory
ww1, ww2 = np.meshgrid(np.linspace(-1, 6, 100), np.linspace(-6, 6, 100))
L_grid   = (ww1 - 2)**2 + 0.5*(ww2 + 3)**2
cs       = axes[1].contourf(ww1, ww2, L_grid, levels=25, cmap="plasma", alpha=.7)
plt.colorbar(cs, ax=axes[1])
axes[1].plot(history[:, 0], history[:, 1], "o-", ms=3, color="white", lw=1, label="trajectory")
axes[1].scatter([2], [-3], marker="*", s=250, color="gold", zorder=5, label="minimum")
axes[1].legend(); axes[1].set_title("Gradient-descent trajectory in loss landscape")
axes[1].set_xlabel("w₁"); axes[1].set_ylabel("w₂")

plt.tight_layout(); plt.show()
Final w  = [ 2.00000005 -2.9982522 ]  (target: [2, -3])
Final L  = 1.53e-06
C:\Users\HOTMAIL\AppData\Local\Temp\ipykernel_13648\2447147685.py:37: UserWarning: Glyph 8321 (\N{SUBSCRIPT ONE}) missing from font(s) Arial.
  plt.tight_layout(); plt.show()
C:\Users\HOTMAIL\AppData\Local\Temp\ipykernel_13648\2447147685.py:37: UserWarning: Glyph 8322 (\N{SUBSCRIPT TWO}) missing from font(s) Arial.
  plt.tight_layout(); plt.show()
C:\DevShared\repo\courses\01_aiml_fundamentals\AIMLStatsPython\.venv\Lib\site-packages\IPython\core\pylabtools.py:170: UserWarning: Glyph 8321 (\N{SUBSCRIPT ONE}) missing from font(s) Arial.
  fig.canvas.print_figure(bytes_io, **kw)
C:\DevShared\repo\courses\01_aiml_fundamentals\AIMLStatsPython\.venv\Lib\site-packages\IPython\core\pylabtools.py:170: UserWarning: Glyph 8322 (\N{SUBSCRIPT TWO}) missing from font(s) Arial.
  fig.canvas.print_figure(bytes_io, **kw)
No description has been provided for this image

Section 11.5 — <code>scipy.optimize</code> with Explicit Gradient

Real optimisation problems are rarely convex quadratics. The Rosenbrock function (also called the "banana function") is a classic non-convex test case:

$$\boxed{f(w_1, w_2) = (1 - w_1)^2 + 100(w_2 - w_1^2)^2}$$

It has a narrow, curved valley and a unique global minimum at $(1, 1)$ that is hard to reach for first-order methods with fixed step sizes. The analytic gradient is:

$$\nabla f = \begin{pmatrix}-2(1-w_1) - 400 w_1(w_2 - w_1^2) \\ 200(w_2 - w_1^2)\end{pmatrix}$$

scipy.optimize.minimize with method="BFGS" (Broyden–Fletcher–Goldfarb– Shanno) is a quasi-Newton method that builds an approximation to the Hessian using successive gradient evaluations. Unlike vanilla gradient descent, BFGS adapts its step to the local curvature and converges superlinearly near the minimum. Providing the analytic Jacobian via jac= makes each step more accurate and roughly halves the number of function evaluations compared to finite-difference gradients.

In [7]:
# === Section 11.5: BFGS on the Rosenbrock function ===
def rosenbrock(w):
    return (1.0 - w[0])**2 + 100.0*(w[1] - w[0]**2)**2

def rosenbrock_grad(w):
    dw0 = -2.0*(1.0 - w[0]) - 400.0*w[0]*(w[1] - w[0]**2)
    dw1 =  200.0*(w[1] - w[0]**2)
    return np.array([dw0, dw1])

w0  = np.array([-1.5, 2.0])
res = minimize(rosenbrock, w0, jac=rosenbrock_grad, method="BFGS",
               options={"gtol": 1e-8})

print(f"Converged   : {res.success}")
print(f"Minimum at  : {res.x}  (target: [1, 1])")
print(f"f(w*)       : {res.fun:.2e}")
print(f"Iterations  : {res.nit}")
print(f"Grad evals  : {res.njev}")

# Visual: loss landscape + final solution
ww1, ww2 = np.meshgrid(np.linspace(-2, 2, 300), np.linspace(-1, 3, 300))
Z = rosenbrock([ww1, ww2])

fig, ax = plt.subplots(figsize=(7, 5))
cs = ax.contourf(ww1, ww2, np.log1p(Z), levels=40, cmap="magma")
plt.colorbar(cs, ax=ax, label="log(1 + f)")
ax.scatter(*res.x, marker="*", s=300, color="gold", zorder=5, label=f"BFGS min {res.x}")
ax.scatter(*w0,    marker="o", s=100, color="cyan",  zorder=5, label=f"start {w0}")
ax.set_title("Rosenbrock landscape  (log scale)"); ax.legend()
ax.set_xlabel("w₁"); ax.set_ylabel("w₂")
plt.tight_layout(); plt.show()
Converged   : True
Minimum at  : [1. 1.]  (target: [1, 1])
f(w*)       : 9.23e-23
Iterations  : 38
Grad evals  : 45
C:\Users\HOTMAIL\AppData\Local\Temp\ipykernel_13648\1834719633.py:31: UserWarning: Glyph 8321 (\N{SUBSCRIPT ONE}) missing from font(s) Arial.
  plt.tight_layout(); plt.show()
C:\Users\HOTMAIL\AppData\Local\Temp\ipykernel_13648\1834719633.py:31: UserWarning: Glyph 8322 (\N{SUBSCRIPT TWO}) missing from font(s) Arial.
  plt.tight_layout(); plt.show()
No description has been provided for this image

Section 11.6 — Summary & Quick Reference

Key Formulas

Concept Formula Python
Gradient $\nabla f = [\partial f/\partial x_1, \ldots, \partial f/\partial x_n]^T$ sp.diff(f, x) / jac=
Jacobian $J_{ij} = \partial F_i / \partial x_j$ F.jacobian([x1, x2])
Hessian $H_{ij} = \partial^2 g / \partial x_i \partial x_j$ sp.hessian(g, vars)
Chain rule $dz/dx = (dz/dy)(dy/dx)$ automatic via autograd/SymPy
Central FD $(f(x+h)-f(x-h))/2h$ manual or scipy.optimize.approx_fprime
GD update $\mathbf{w} \leftarrow \mathbf{w} - \eta\nabla\mathcal{L}$ w -= lr * grad(w)
BFGS quasi-Newton, superlinear convergence minimize(f, w0, jac=g, method="BFGS")

ML Connections

Concept Where It Appears
Gradient Gradient descent, AdaGrad, RMSProp, Adam — all follow $-\nabla\mathcal{L}$
Jacobian Backward pass of any layer: $\delta\mathbf{x} = \mathbf{J}^T\,\delta\mathbf{y}$
Hessian Newton's method, natural gradient, Fisher information matrix
Chain rule Backpropagation — the chain rule applied through the computation graph
FD gradient check Unit-testing custom backward implementations
BFGS / L-BFGS Fine-tuning, hyperparameter optimisation, second-order training

Optimizer Comparison

Method Info used Convergence Memory Use when
Gradient Descent $\nabla\mathcal{L}$ Linear $O(n)$ Simple demos, convex problems
SGD + momentum $\nabla\mathcal{L}$ mini-batch Sub-linear $O(n)$ Large neural nets
Adam adaptive per-param Fast in practice $O(n)$ Default for deep learning
BFGS $\nabla\mathcal{L}$ + approx $\mathbf{H}^{-1}$ Superlinear $O(n^2)$ Small parameter spaces
L-BFGS limited-memory BFGS Superlinear $O(mn)$ Medium problems, fine-tuning

What's Next?

$$\boxed{\text{Ch 11: Calculus}} \;\to\; \boxed{\text{Ch 12: Scientific Python}} \;\to\; \boxed{\text{Ch 13: Information Theory}} \;\to\; \boxed{\text{Ch 14: Neural Networks}}$$

Chapter 12 shifts from mathematics to engineering: how do you make the computations you just learned run 100× faster? Vectorisation, broadcasting, and advanced NumPy patterns turn the calculus of this chapter into production-ready code.


References

  1. Spivak, M. (1965). Calculus on Manifolds. W. A. Benjamin. The definitive rigorous treatment of multivariable calculus and the generalised chain rule.

  2. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 4 (Numerical Computation) and Chapter 6 (Deep Feedforward Networks — backpropagation derivation). https://www.deeplearningbook.org/

  3. Nocedal, J., & Wright, S. J. (2006). Numerical Optimization (2nd ed.). Springer. The graduate reference for gradient descent, quasi-Newton methods (BFGS), and line-search convergence theory.

  4. Bishop, C. M., & Bishop, H. (2024). Deep Learning: Foundations and Concepts. Springer. Chapter 7 covers gradient-based optimisation in the modern deep-learning context. https://www.bishopbook.com/

  5. Wikipedia. Gradient. https://en.wikipedia.org/wiki/Gradient

  6. Wikipedia. Chain rule. https://en.wikipedia.org/wiki/Chain_rule

  7. Wikipedia. Backpropagation. https://en.wikipedia.org/wiki/Backpropagation

  8. Wikipedia. Broyden–Fletcher–Goldfarb–Shanno algorithm. https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm

  9. SciPy Development Team. scipy.optimize.minimize documentation. https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html

  10. SymPy Development Team. SymPy calculus reference. https://docs.sympy.org/latest/modules/calculus/