Home / About Us / Volume I Contents / Ch 14: Neural Networks

Statistical Basis of AI/ML — Neural Networks

Chapter 14 — Neurons, Layers, and the Backpropagation Algorithm


Why This Chapter?

In Chapter 10 you established that every linear layer of a network is a matrix multiplication — a change of basis in feature space. In Chapter 11 you derived gradient descent and watched Adam navigate a curved loss surface. In Chapter 13 you learned why cross-entropy is the canonical loss for classifiers, and how the combined sigmoid-plus-cross-entropy gradient simplifies to $\hat{p} - y$. This chapter assembles those pieces into a complete neural network and shows you exactly where each piece fits.

A neural network is, at its core, a composition of linear maps interleaved with non-linear activation functions. The linear maps provide the representational capacity; the activations are what prevent the entire stack from collapsing into a single matrix multiplication. Understanding why we need non-linearity — and which non-linearity to use — is the first practical skill this chapter builds.

The second skill is backpropagation: the recursive application of the chain rule that propagates loss gradients from the output layer back to every weight in the network. Backprop is often treated as a black box inside deep-learning frameworks, but implementing it from scratch in NumPy removes all mystery. You will see exactly why a ReLU network does not suffer from the vanishing-gradient pathology that crippled sigmoid-only networks in the 1990s.

The chapter closes by handing off to scikit-learn’s MLPClassifier — a fully-featured implementation with Adam optimisation, L2 regularisation, and early stopping — and applies it to the Breast Cancer Wisconsin dataset. By the end of this chapter you will be able to write, train, and evaluate a feedforward network both from scratch and with production-grade tooling.

§ Topic Core concept
14.0 Setup Imports and configuration
14.1 The Perceptron Linear threshold units; AND vs XOR
14.2 Activation Functions Sigmoid, ReLU, tanh; vanishing gradient
14.3 The Forward Pass Layer notation; matrix form; NumPy implementation
14.4 Backpropagation Chain rule through layers; gradient check
14.5 Training Loop Mini-batch GD from scratch on make_moons
14.6 sklearn MLPClassifier Adam, L2 regularisation; breast-cancer dataset
14.7 Summary Key formulas, ML connections, What’s Next

Section 14.0 — Setup: Imports & Configuration

Library Role in This Chapter
NumPy All from-scratch network arithmetic: matrix multiply, activation functions, backprop
Matplotlib / Seaborn Decision boundaries, activation-function galleries, loss curves
scikit-learn MLPClassifier, make_moons, load_breast_cancer, preprocessing

The random number generator is seeded once and reused throughout so that every figure is fully reproducible.

In [1]:
# === Section 14.0: Imports & configuration ===
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.neural_network import MLPClassifier
from sklearn.datasets import load_breast_cancer, make_moons
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, accuracy_score

rng = np.random.default_rng(42)
sns.set_theme(style="whitegrid", palette="muted")

import numpy, sklearn
print(f"NumPy {numpy.__version__}  |  scikit-learn {sklearn.__version__}")
print("Chapter 14 — Neural Networks: environment ready.")
NumPy 2.3.5  |  scikit-learn 1.8.0
Chapter 14 — Neural Networks: environment ready.

Section 14.1 — The Neuron and the Perceptron

The biological metaphor is useful but loose: a neuron receives weighted inputs, sums them, and fires if the total exceeds a threshold. The mathematical abstraction is the perceptron proposed by Rosenblatt (1958):

$$\boxed{\hat{y} = \text{sign}(\mathbf{w}^\top \mathbf{x} + b)}$$

where $\mathbf{w} \in \mathbb{R}^d$ is a weight vector, $b$ is a scalar bias, and $\text{sign}$ returns $+1$ if the argument is non-negative and $-1$ otherwise. The perceptron draws a single hyperplane through the input space, so it can only separate linearly separable classes.

The XOR problem. Minsky & Papert (1969) showed that XOR — the canonical non-linearly-separable function — cannot be computed by a single perceptron. This observation suppressed neural-network research for a decade until multi-layer networks and the backpropagation algorithm were formalised in the 1980s. The solution to XOR requires at least one hidden layer.

In the code cell below we implement the Heaviside step activation, confirm that a perceptron with $\mathbf{w}=[1,1]^\top$, $b=-1.5$ correctly computes AND, and verify that no single-layer weight setting can solve XOR.

In [2]:
# === Section 14.1: Perceptron on AND and XOR ===

def step(z):
    """Heaviside step activation."""
    return (z >= 0).astype(float)

X_logic = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y_and   = np.array([0, 0, 0, 1], dtype=float)
y_xor   = np.array([0, 1, 1, 0], dtype=float)

# AND gate: threshold at 1.5
w_and, b_and = np.array([1.0, 1.0]), -1.5
pred_and = step(X_logic @ w_and + b_and)
print("AND gate — predictions :", pred_and.astype(int))
print("AND gate — correct?    :", np.all(pred_and == y_and))

# XOR: try w=[1,1], b=-0.5  (threshold at 0.5)
w_xor, b_xor = np.array([1.0, 1.0]), -0.5
pred_xor = step(X_logic @ w_xor + b_xor)
print("\nXOR attempt — predictions:", pred_xor.astype(int))
print("XOR attempt — correct?    :", np.all(pred_xor == y_xor))
print("(No single linear boundary can separate XOR — this confirms Minsky-Papert.)")

# Visualise
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
configs = [
    (y_and, "AND gate (linearly separable)"),
    (y_xor, "XOR gate (not linearly separable)"),
]
for ax, (y, title) in zip(axes, configs):
    for xi, yi in zip(X_logic, y):
        ax.scatter(xi[0], xi[1],
                   marker="o" if yi == 1 else "s",
                   s=220,
                   color="steelblue" if yi == 1 else "tomato",
                   zorder=3,
                   label=("class 1" if yi == 1 else "class 0"))
    xx = np.linspace(-0.3, 1.3, 200)
    ax.plot(xx, 1.5 - xx, "k--", lw=1.5, label="boundary $x_1+x_2=1.5$")
    ax.set_xlim(-0.3, 1.3); ax.set_ylim(-0.3, 1.3)
    ax.set_xlabel("$x_1$"); ax.set_ylabel("$x_2$")
    ax.set_title(title)
    handles, labels = ax.get_legend_handles_labels()
    unique = dict(zip(labels, handles))
    ax.legend(unique.values(), unique.keys(), fontsize=8)

plt.suptitle("Perceptron decision boundaries"); plt.tight_layout(); plt.show()
AND gate — predictions : [0 0 0 1]
AND gate — correct?    : True

XOR attempt — predictions: [0 1 1 1]
XOR attempt — correct?    : False
(No single linear boundary can separate XOR — this confirms Minsky-Papert.)
No description has been provided for this image

Section 14.2 — Activation Functions: The Engine of Non-Linearity

Without non-linear activations, stacking multiple linear layers collapses to a single affine map: $\mathbf{W}_2(\mathbf{W}_1 \mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2 = (\mathbf{W}_2\mathbf{W}_1)\mathbf{x} + \text{const.}$ Non-linearity is what gives deep networks their expressive power.

The four activations covered here form the practical vocabulary of feedforward networks:

Activation Formula Derivative Typical use
Sigmoid $\sigma$ $1/(1+e^{-z})$ $\sigma(1-\sigma)$ Output layer for binary classification
ReLU $\max(0, z)$ $\mathbf{1}[z>0]$ Hidden layers — default choice
Tanh $(e^z - e^{-z})/(e^z + e^{-z})$ $1 - \tanh^2 z$ RNNs; zero-centred
Leaky ReLU $z$ if $z>0$ else $\alpha z$ $1$ or $\alpha$ Avoids dead neurons

Vanishing gradient. Sigmoid and tanh both saturate: their derivatives approach zero for large $|z|$. In a 10-layer sigmoid network, the gradient reaching layer 1 is the product of 10 near-zero terms — effectively zero. ReLU has a derivative of exactly 1 for any positive input, so gradients propagate without attenuation through the active units. This is the primary reason ReLU replaced sigmoid in deep hidden layers after Glorot & Bengio (2010).

$$\boxed{\text{ReLU}(z) = \max(0,\, z), \qquad \frac{d}{dz}\text{ReLU}(z) = \mathbf{1}[z > 0]}$$

In [3]:
# === Section 14.2: Activation functions and derivatives ===

def sigmoid(z):            return 1 / (1 + np.exp(-z))
def relu(z):               return np.maximum(0, z)
def tanh_act(z):           return np.tanh(z)
def leaky_relu(z, a=0.1):  return np.where(z >= 0, z, a * z)

def d_sigmoid(z):          s = sigmoid(z); return s * (1 - s)
def d_relu(z):             return (z > 0).astype(float)
def d_tanh(z):             return 1 - np.tanh(z) ** 2
def d_leaky(z, a=0.1):    return np.where(z >= 0, 1.0, a)

z_vals = np.linspace(-5, 5, 400)
configs = [
    ("Sigmoid $\\sigma$",   sigmoid,    d_sigmoid, "steelblue"),
    ("ReLU",               relu,       d_relu,    "tomato"),
    ("Tanh",               tanh_act,   d_tanh,    "seagreen"),
    ("Leaky ReLU",         leaky_relu, d_leaky,   "darkorange"),
]

fig, axes = plt.subplots(2, 4, figsize=(16, 6))
for col, (name, f, df, color) in enumerate(configs):
    axes[0, col].plot(z_vals, f(z_vals), color=color, lw=2)
    axes[0, col].set_title(name)
    axes[0, col].axhline(0, color="gray", lw=0.5)
    axes[0, col].axvline(0, color="gray", lw=0.5)
    axes[0, col].set_ylim(-1.3, 1.3)

    axes[1, col].plot(z_vals, df(z_vals), color=color, lw=2, ls="--")
    axes[1, col].set_title(f"$d/dz$ {name}")
    axes[1, col].axhline(0, color="gray", lw=0.5)
    axes[1, col].set_ylim(-0.4, 1.3)

axes[0, 0].set_ylabel("$g(z)$")
axes[1, 0].set_ylabel("$g'(z)$")
plt.suptitle("Activation functions and their derivatives", y=1.01)
plt.tight_layout(); plt.show()

print("Vanishing gradient demo:")
print(f"  Sigmoid derivative at z=5 : {d_sigmoid(5.0):.7f}  <- near zero")
print(f"  ReLU    derivative at z=5 : {d_relu(np.float64(5.0)):.7f}  <- exactly 1")
No description has been provided for this image
Vanishing gradient demo:
  Sigmoid derivative at z=5 : 0.0066481  <- near zero
  ReLU    derivative at z=5 : 1.0000000  <- exactly 1

Section 14.3 — Multi-Layer Architecture and the Forward Pass

A feedforward network with $L$ layers maps an input $\mathbf{a}^{[0]} = \mathbf{x}$ through a sequence of affine-then-activate steps. Using superscript $[l]$ to index the layer:

$$\mathbf{z}^{[l]} = \mathbf{W}^{[l]}\mathbf{a}^{[l-1]} + \mathbf{b}^{[l]}, \qquad \mathbf{a}^{[l]} = g^{[l]}(\mathbf{z}^{[l]})$$

For a mini-batch of $m$ examples the columns of $\mathbf{A}^{[l]}$ are the activations of all $m$ examples at layer $l$ simultaneously. Each weight matrix $\mathbf{W}^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}$ connects $n_{l-1}$ input neurons to $n_l$ output neurons.

He initialisation. For ReLU networks, weights are drawn from $\mathcal{N}\!\left(0,\, \sqrt{2/n_{l-1}}\right)$ (He et al., 2015). This keeps the variance of activations stable across layers at initialisation, preventing the gradient signal from exploding or vanishing before a single update step.

$$\boxed{\mathbf{W}^{[l]} \sim \mathcal{N}\!\left(0,\; \sqrt{\tfrac{2}{n_{l-1}}}\right), \quad \mathbf{b}^{[l]} = \mathbf{0}}$$

In [4]:
# === Section 14.3: He initialisation and the forward pass ===

def init_params(layer_dims, seed=42):
    """He initialisation for ReLU networks.

    Parameters
    ----------
    layer_dims : list[int]
        Number of neurons per layer, including input (index 0) and output (last).

    Returns
    -------
    dict  --  keys 'W1', 'b1', 'W2', 'b2', ...
    """
    rng_loc = np.random.default_rng(seed)
    params = {}
    for l in range(1, len(layer_dims)):
        n_in  = layer_dims[l - 1]
        n_out = layer_dims[l]
        params[f"W{l}"] = rng_loc.standard_normal((n_out, n_in)) * np.sqrt(2.0 / n_in)
        params[f"b{l}"] = np.zeros((n_out, 1))
    return params


def forward_pass(X, params, n_layers):
    """Forward pass through a (n_layers-1)-hidden-layer network.

    Parameters
    ----------
    X        : ndarray, shape (n_features, n_samples)
    params   : dict from init_params
    n_layers : int  -- len(layer_dims)

    Returns
    -------
    A_out : ndarray  -- output activations
    cache : dict     -- {A0, Z1, A1, Z2, A2, ...} needed for backprop
    """
    cache = {"A0": X}
    A = X
    for l in range(1, n_layers):
        Z = params[f"W{l}"] @ A + params[f"b{l}"]
        # Hidden layers use ReLU; output layer uses sigmoid
        A = relu(Z) if l < n_layers - 1 else sigmoid(Z)
        cache[f"Z{l}"] = Z
        cache[f"A{l}"] = A
    return A, cache


# Sanity check: [3 features] -> [4 hidden] -> [1 output]
n_feat = 3
X_toy  = rng.standard_normal((n_feat, 5))
dims_toy = [n_feat, 4, 1]
params_toy = init_params(dims_toy)

A_out, cache_toy = forward_pass(X_toy, params_toy, len(dims_toy))
print("Input  shape:", X_toy.shape)
print("Hidden shape:", cache_toy["A1"].shape)
print("Output shape:", A_out.shape)
print("Output (probabilities):\n", A_out.round(4))
print("\nAll outputs in (0, 1):", bool(np.all((A_out > 0) & (A_out < 1))))
Input  shape: (3, 5)
Hidden shape: (4, 5)
Output shape: (1, 5)
Output (probabilities):
 [[0.7846 0.312  0.707  0.5108 0.353 ]]

All outputs in (0, 1): True

Section 14.4 — Loss Functions and Backpropagation

For binary classification we use the binary cross-entropy loss introduced in Chapter 13:

$$\mathcal{L} = -\frac{1}{m}\sum_{i=1}^{m}\left[y_i\log\hat{p}_i + (1-y_i)\log(1-\hat{p}_i)\right]$$

Backpropagation applies the chain rule recursively from layer $L$ to layer $1$. The key identity — derived in Chapter 13 — is that the combined gradient of cross-entropy through a sigmoid output simplifies beautifully:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{z}^{[L]}} = \mathbf{a}^{[L]} - \mathbf{y}$$

For a hidden layer $l$ with ReLU:

$$\delta^{[l]} = \left(\mathbf{W}^{[l+1]\top}\delta^{[l+1]}\right) \odot \mathbf{1}\!\left[\mathbf{z}^{[l]}>0\right]$$

The weight gradients follow from the outer product of the error signal and the previous layer’s activations:

$$\boxed{\frac{\partial \mathcal{L}}{\partial \mathbf{W}^{[l]}} = \frac{1}{m}\,\delta^{[l]}\,\mathbf{A}^{[l-1]\top}, \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{b}^{[l]}} = \frac{1}{m}\sum_i \delta^{[l]}_i}$$

We verify our implementation with a finite-difference gradient check: the analytical gradient must agree with $(\mathcal{L}(w+\epsilon) - \mathcal{L}(w-\epsilon)) / 2\epsilon$ to at least five significant figures.

In [5]:
# === Section 14.4: Cross-entropy loss and backpropagation ===

def compute_loss(Y_hat, Y, eps=1e-12):
    """Binary cross-entropy.  Y_hat, Y: (1, n_samples)."""
    return -float(np.mean(
        Y * np.log(Y_hat + eps) + (1 - Y) * np.log(1 - Y_hat + eps)
    ))


def backward_pass(X, Y, params, cache, n_layers):
    """Backpropagation for a ReLU-hidden / sigmoid-output network.

    Returns
    -------
    grads : dict  --  keys 'dW1', 'db1', 'dW2', 'db2', ...
    """
    m = X.shape[1]
    grads = {}
    # Combined CE + sigmoid gradient at the output layer
    dA = cache[f"A{n_layers - 1}"] - Y
    for l in range(n_layers - 1, 0, -1):
        A_prev = cache[f"A{l - 1}"]
        W      = params[f"W{l}"]
        if l == n_layers - 1:
            dZ = dA                              # output: dL/dz = a - y
        else:
            dZ = dA * (cache[f"Z{l}"] > 0)     # ReLU derivative
        grads[f"dW{l}"] = (dZ @ A_prev.T) / m
        grads[f"db{l}"] = np.mean(dZ, axis=1, keepdims=True)
        dA = W.T @ dZ
    return grads


# Gradient check on a tiny XOR-like dataset
X_bp  = np.array([[0, 0, 1, 1], [0, 1, 0, 1]], dtype=float)
Y_bp  = np.array([[0, 1, 1, 0]], dtype=float)
p_bp  = init_params([2, 4, 1])
out_bp, cache_bp = forward_pass(X_bp, p_bp, 3)
grads = backward_pass(X_bp, Y_bp, p_bp, cache_bp, 3)

print(f"Loss before any update: {compute_loss(out_bp, Y_bp):.4f}")
print(f"dW1 shape: {grads['dW1'].shape}   dW2 shape: {grads['dW2'].shape}")

# Finite-difference check on W2[0, 0]
eps_fd = 1e-5
p_plus  = {k: v.copy() for k, v in p_bp.items()}
p_minus = {k: v.copy() for k, v in p_bp.items()}
p_plus["W2"][0, 0]  += eps_fd
p_minus["W2"][0, 0] -= eps_fd
o_plus,  _ = forward_pass(X_bp, p_plus,  3)
o_minus, _ = forward_pass(X_bp, p_minus, 3)

fd_grad = (compute_loss(o_plus, Y_bp) - compute_loss(o_minus, Y_bp)) / (2 * eps_fd)
an_grad = float(grads["dW2"][0, 0])
rel_err = abs(fd_grad - an_grad) / (abs(fd_grad) + abs(an_grad) + 1e-12)

print(f"\nGradient check — W2[0, 0]:")
print(f"  Finite-difference : {fd_grad:.8f}")
print(f"  Analytical (backprop): {an_grad:.8f}")
print(f"  Relative error    : {rel_err:.2e}")
assert rel_err < 1e-4, f"Gradient check failed: rel_err={rel_err:.2e}"
print("[OK] Gradient check passed.")
Loss before any update: 0.7306
dW1 shape: (4, 2)   dW2 shape: (1, 4)

Gradient check — W2[0, 0]:
  Finite-difference : -0.04535058
  Analytical (backprop): -0.04535058
  Relative error    : 3.06e-11
[OK] Gradient check passed.

Section 14.5 — Training: Gradient Descent and the Full Loop

Training assembles the forward pass, loss computation, backprop, and a parameter update into a loop over epochs (full passes through the data). We use plain mini-batch gradient descent with a fixed learning rate:

$$\mathbf{W}^{[l]} \leftarrow \mathbf{W}^{[l]} - \eta\,\frac{\partial\mathcal{L}}{\partial\mathbf{W}^{[l]}}, \qquad \mathbf{b}^{[l]} \leftarrow \mathbf{b}^{[l]} - \eta\,\frac{\partial\mathcal{L}}{\partial\mathbf{b}^{[l]}}$$

For this demonstration we use the full batch (all 300 training examples each epoch) for simplicity. The dataset is make_moons with added noise — a crescent-shaped binary problem that is not linearly separable, so it requires at least one hidden layer. We train a $[2 \to 8 \to 8 \to 1]$ network for 2 000 epochs and visualise the learned decision boundary alongside the loss curve.

Regularisation. We omit explicit weight decay in this scratch implementation to keep the code readable. sklearn’s MLPClassifier in §14.6 adds L2 regularisation ($\alpha$) and early stopping automatically.

In [6]:
# === Section 14.5: Training loop on make_moons ===

def train_network(X, Y, layer_dims, lr=0.05, epochs=2000, print_every=500):
    """Train a feedforward network; return (params, loss_history)."""
    params = init_params(layer_dims)
    n = len(layer_dims)
    history = []
    for epoch in range(epochs + 1):
        A_out, cache = forward_pass(X, params, n)
        loss = compute_loss(A_out, Y)
        history.append(loss)
        grads = backward_pass(X, Y, params, cache, n)
        for l in range(1, n):
            params[f"W{l}"] -= lr * grads[f"dW{l}"]
            params[f"b{l}"] -= lr * grads[f"db{l}"]
        if epoch % print_every == 0:
            acc = float(np.mean((A_out >= 0.5) == Y))
            print(f"  Epoch {epoch:5d}  loss={loss:.4f}  train_acc={acc:.3f}")
    return params, history


# Dataset
X_moons, y_moons = make_moons(n_samples=400, noise=0.20, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(
    X_moons, y_moons, test_size=0.25, random_state=42
)
scaler = StandardScaler().fit(X_tr)
X_tr_s = scaler.transform(X_tr).T           # (2, 300)
X_te_s = scaler.transform(X_te).T           # (2, 100)
Y_tr   = y_tr.reshape(1, -1).astype(float)
Y_te   = y_te.reshape(1, -1).astype(float)

print("Training [2 -> 8 -> 8 -> 1] network on make_moons:")
params_m, loss_hist = train_network(
    X_tr_s, Y_tr, [2, 8, 8, 1], lr=0.1, epochs=2000
)

# Test accuracy
A_te, _ = forward_pass(X_te_s, params_m, 4)
acc_te  = float(np.mean((A_te >= 0.5) == Y_te))
print(f"\nTest accuracy: {acc_te:.3f}")

# ---- Decision boundary + loss curve ----
fig, axes = plt.subplots(1, 2, figsize=(13, 5))

# Decision surface
h = 0.03
x0_min, x0_max = X_moons[:, 0].min() - 0.5, X_moons[:, 0].max() + 0.5
x1_min, x1_max = X_moons[:, 1].min() - 0.5, X_moons[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.arange(x0_min, x0_max, h),
                     np.arange(x1_min, x1_max, h))
grid_raw = np.c_[xx.ravel(), yy.ravel()]
grid_s   = scaler.transform(grid_raw).T
Z_grid, _ = forward_pass(grid_s, params_m, 4)
Z_grid    = Z_grid.reshape(xx.shape)

axes[0].contourf(xx, yy, Z_grid, levels=50, cmap="RdBu", alpha=0.6)
axes[0].scatter(X_moons[:, 0], X_moons[:, 1],
                c=y_moons, cmap="RdBu", edgecolors="k", s=30)
axes[0].set_title(f"Learned boundary  (test acc = {acc_te:.2f})")
axes[0].set_xlabel("$x_1$"); axes[0].set_ylabel("$x_2$")

# Loss curve
axes[1].plot(loss_hist, color="steelblue", lw=1.5)
axes[1].set_xlabel("Epoch")
axes[1].set_ylabel("Binary cross-entropy loss")
axes[1].set_title("Training loss curve")

plt.suptitle('From-scratch neural net on make_moons'); plt.tight_layout(); plt.show()
Training [2 -> 8 -> 8 -> 1] network on make_moons:
  Epoch     0  loss=0.6483  train_acc=0.510
  Epoch   500  loss=0.1971  train_acc=0.893
  Epoch  1000  loss=0.1102  train_acc=0.963
  Epoch  1500  loss=0.0721  train_acc=0.973
  Epoch  2000  loss=0.0602  train_acc=0.980

Test accuracy: 0.960
No description has been provided for this image

Section 14.6 — From Scratch to sklearn: MLPClassifier

Our from-scratch implementation proved the algorithm. For production work, scikit-learn’s MLPClassifier adds:

Feature Detail
Solver adam (adaptive moment estimation, see Chapter 11)
L2 regularisation alpha parameter adds $\frac{\alpha}{2}\|\mathbf{W}\|_F^2$ to the loss
Early stopping Holds out a validation fraction; stops when validation score stagnates
Batch size Default min(200, n_samples) for adam

We apply it to the Breast Cancer Wisconsin dataset (569 samples, 30 features, binary malignant/benign labels) — the same dataset used in Chapter 7 for SVM and logistic regression, so results are directly comparable.

Always standardise features before training a neural network: the weight initialisation assumes inputs of order $\mathcal{O}(1)$, and un-scaled features with very different magnitudes cause the gradients for different weights to be on completely different scales, severely slowing convergence.

In [7]:
# === Section 14.6: sklearn MLPClassifier on Breast Cancer Wisconsin ===

data_bc = load_breast_cancer()
X_bc, y_bc = data_bc.data, data_bc.target

X_tr_bc, X_te_bc, y_tr_bc, y_te_bc = train_test_split(
    X_bc, y_bc, test_size=0.20, random_state=42, stratify=y_bc
)
sc_bc   = StandardScaler().fit(X_tr_bc)
X_tr_sc = sc_bc.transform(X_tr_bc)
X_te_sc = sc_bc.transform(X_te_bc)

mlp = MLPClassifier(
    hidden_layer_sizes=(64, 32),
    activation="relu",
    solver="adam",
    alpha=1e-4,               # L2 weight-decay
    max_iter=500,
    random_state=42,
    early_stopping=True,
    validation_fraction=0.15,
    n_iter_no_change=20,
)
mlp.fit(X_tr_sc, y_tr_bc)

acc_bc = accuracy_score(y_te_bc, mlp.predict(X_te_sc))
print(f"Test accuracy : {acc_bc:.4f}")
print(f"Stopped after : {mlp.n_iter_} iterations")
print()
print(classification_report(
    y_te_bc, mlp.predict(X_te_sc), target_names=data_bc.target_names
))

# Loss + validation accuracy curves
fig, ax1 = plt.subplots(figsize=(9, 4))
ax1.plot(mlp.loss_curve_, color="steelblue", lw=2, label="Training loss")
ax1.set_xlabel("Iteration")
ax1.set_ylabel("Loss", color="steelblue")
ax1.tick_params(axis="y", labelcolor="steelblue")

if hasattr(mlp, "validation_scores_") and mlp.validation_scores_ is not None:
    ax2 = ax1.twinx()
    ax2.plot(mlp.validation_scores_, color="tomato", lw=2, ls="--",
             label="Validation accuracy")
    ax2.set_ylabel("Validation accuracy", color="tomato")
    ax2.tick_params(axis="y", labelcolor="tomato")
    ax2.set_ylim(0.85, 1.0)

ax1.set_title("MLPClassifier — Breast Cancer Wisconsin")
lines1, labels1 = ax1.get_legend_handles_labels()
if hasattr(mlp, "validation_scores_") and mlp.validation_scores_ is not None:
    lines2, labels2 = ax2.get_legend_handles_labels()
    ax1.legend(lines1 + lines2, labels1 + labels2, loc="center right")
else:
    ax1.legend(loc="upper right")
plt.tight_layout(); plt.show()
Test accuracy : 0.9737
Stopped after : 56 iterations

              precision    recall  f1-score   support

   malignant       0.95      0.98      0.96        42
      benign       0.99      0.97      0.98        72

    accuracy                           0.97       114
   macro avg       0.97      0.97      0.97       114
weighted avg       0.97      0.97      0.97       114

No description has been provided for this image

Section 14.7 — Summary & Quick Reference

Key Formulas

Concept Formula Where it appeared
Forward pass $\mathbf{z}^{[l]} = \mathbf{W}^{[l]}\mathbf{a}^{[l-1]}+\mathbf{b}^{[l]}$ §14.3
ReLU $g(z)=\max(0,z),\; g'(z)=\mathbf{1}[z>0]$ §14.2
Sigmoid $\sigma(z)=1/(1+e^{-z}),\; \sigma'=\sigma(1-\sigma)$ §14.2, Ch13
CE loss $\mathcal{L}=-\frac{1}{m}\sum[y\log\hat{p}+(1-y)\log(1-\hat{p})]$ §14.4, Ch13
Output delta $\delta^{[L]}=\hat{\mathbf{p}}-\mathbf{y}$ §14.4
Hidden delta $\delta^{[l]}=(\mathbf{W}^{[l+1]\top}\delta^{[l+1]})\odot\mathbf{1}[z^{[l]}>0]$ §14.4
Weight gradient $\partial\mathcal{L}/\partial\mathbf{W}^{[l]}=\delta^{[l]}\mathbf{A}^{[l-1]\top}/m$ §14.4
He init $w\sim\mathcal{N}(0,\sqrt{2/n_{l-1}})$ §14.3

ML Connections

Concept Where it appears in ML
Matrix multiply Every linear layer — GPU-optimised BLAS routines (Ch10)
Gradient descent / Adam Training loop in §14.5; production use in sklearn (Ch11)
Cross-entropy Classification loss for every neural network (Ch13)
ReLU hidden + sigmoid output Standard feedforward classifier architecture
L2 regularisation sklearn alpha; equivalent to MAP with Gaussian weight prior
Early stopping Validation-set based; prevents overfitting without explicit penalty
Universal approximation Any continuous function can be approximated by a wide enough 1-hidden-layer ReLU net (Cybenko 1989; Hornik 1991)

What’s Next?

$$\boxed{\text{Ch14: Neural Networks} \;\longrightarrow\; \text{Convolutional, Recurrent, and Attention-based Architectures}}$$

This chapter covered the feedforward (fully-connected) network — the foundation of deep learning. The next natural extensions are:

  • Convolutional Neural Networks (CNNs): replace dense weight matrices with shared convolutional filters to exploit spatial structure in images.
  • Recurrent Neural Networks (RNNs) / LSTMs: fold a hidden state back into the next time step to process sequences of variable length.
  • Transformers: replace recurrence with self-attention — the architecture underlying every large language model, including GPT-4 and the model delivering this book.

All three architectures share the same core ingredients developed here: forward passes composed of matrix multiplications, non-linear activations, and backpropagation through the chain rule.


References

  1. Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organisation in the brain. Psychological Review, 65(6), 386–408. The original perceptron paper.

  2. Minsky, M., & Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry. MIT Press. Proved the XOR limitation of single-layer perceptrons.

  3. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. https://doi.org/10.1038/323533a0

  4. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4), 303–314. Universal approximation theorem for sigmoid networks.

  5. Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. AISTATS 2010. Diagnoses vanishing gradients; introduces Xavier initialisation. http://proceedings.mlr.press/v9/glorot10a.html

  6. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. ICCV 2015. Derives He initialisation for ReLU networks. https://arxiv.org/abs/1502.01852

  7. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521, 436–444. https://doi.org/10.1038/nature14539

  8. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapters 6–8 cover feedforward networks, backpropagation, and regularisation. https://www.deeplearningbook.org/

  9. Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimisation. ICLR 2015. https://arxiv.org/abs/1412.6980

  10. Pedregosa, F. et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830. https://jmlr.org/papers/v12/pedregosa11a.html Source for MLPClassifier used in §14.6.