Home / About Us / Volume I Contents / Ch 15: CNNs and PyTorch

Statistical Basis of AI/ML — Convolutional Neural Networks and PyTorch

Chapter 15 — Spatial Structure, Parameter Sharing, and the PyTorch Framework


Why This Chapter?

Chapter 14 built a fully-connected network from NumPy arithmetic and proved that the forward pass, cross-entropy loss, and backpropagation are nothing but matrix calculus (Chapters 10–11) applied in sequence. That from-scratch implementation was instructive, but it has a fundamental limitation: dense layers treat every pixel as an independent feature, discarding the spatial structure that makes images meaningful.

This chapter introduces two ideas that unlock modern computer vision:

  1. The convolution operation — a local, weight-shared linear transform that detects the same edge or texture pattern anywhere in an image, using a fraction of the parameters a dense layer would need.
  2. PyTorch — the numerical framework that replaces hand-rolled NumPy for everything from here forward. PyTorch's autograd engine computes every gradient automatically; its CUDA backend moves computation to the GPU with a single .to(device) call.

By the end of this chapter you will have trained a convolutional network on a real image dataset, watched the GPU absorb the computation, and seen how pretrained ResNet-18 can be fine-tuned to a new task in minutes — the practical workflow for clinical AI applications.

Chapter Contents

Section Title
§15.0 Setup: PyTorch, GPU Verification, and Torchvision
§15.1 The Convolution Operation
§15.2 Pooling, Strides, and Receptive Fields
§15.3 Building a CNN with nn.Module — PyTorch Replaces NumPy
§15.4 Training on MNIST: the Full PyTorch Loop
§15.5 Transfer Learning with ResNet-18
§15.6 Summary & Quick Reference

Section 15.0 — Setup: PyTorch, GPU Verification, and Torchvision

Library Role in this chapter
torch Tensor operations, autograd, GPU compute
torch.nn Layer primitives (Conv2d, MaxPool2d, Linear)
torch.optim Optimisers (SGD, Adam)
torchvision Standard image datasets and pretrained models
matplotlib / seaborn Visualisation
numpy Bridge between PyTorch tensors and NumPy for plotting
In [1]:
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import seaborn as sns
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms, models
from sklearn.metrics import classification_report

sns.set_theme(style="whitegrid", palette="muted")
torch.manual_seed(42)
rng = np.random.default_rng(42)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

print(f"PyTorch  : {torch.__version__}")
print(f"Device   : {device}")
if device.type == "cuda":
    print(f"GPU      : {torch.cuda.get_device_name(0)}")
    print(f"VRAM     : {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
print("Chapter 15 -- CNNs and PyTorch: environment ready.")
PyTorch  : 2.11.0+cu128
Device   : cuda
GPU      : NVIDIA GeForce RTX 5080
VRAM     : 17.1 GB
Chapter 15 -- CNNs and PyTorch: environment ready.

Section 15.1 — The Convolution Operation

From Dense Layers to Filters

In Chapter 14, every hidden unit connected to every input pixel. For a modest $128\times128$ greyscale image, that is $128^2 = 16{,}384$ inputs — so a single hidden layer of 256 units requires $\approx 4.2\text{ M}$ parameters, before seeing a single image.

A convolutional filter (or kernel) of size $k\times k$ instead learns to detect one pattern — an edge, a curve, a texture — and slides that same set of $k^2$ weights across every spatial location. The result is a feature map:

$$(\mathbf{X} \star \mathbf{K})_{i,j} = \sum_{p=0}^{k-1}\sum_{q=0}^{k-1} X_{i+p,\;j+q}\cdot K_{p,q}$$

where $\mathbf{X}$ is the input patch and $\mathbf{K}$ is the filter.

Why this matters clinically

A filter trained to detect a tumour margin learns that pattern once and applies it everywhere in the whole-slide image — regardless of where the margin falls on the slide. Dense layers cannot share this knowledge; every spatial position has its own independent weights.

Dimension arithmetic

For an input of height $H$, width $W$, filter size $k$, padding $p$, and stride $s$:

$$H_{\text{out}} = \left\lfloor \frac{H + 2p - k}{s} \right\rfloor + 1, \qquad W_{\text{out}} = \left\lfloor \frac{W + 2p - k}{s} \right\rfloor + 1$$

With $k=3$, $p=1$, $s=1$ (the most common setting): $H_{\text{out}} = H$ — the spatial size is preserved.

In [2]:
# ── Manual 2-D convolution demo ──────────────────────────────────────────
def conv2d_manual(X: np.ndarray, K: np.ndarray) -> np.ndarray:
    """Slide kernel K over image X (no padding, stride 1)."""
    kh, kw = K.shape
    oh, ow = X.shape[0] - kh + 1, X.shape[1] - kw + 1
    out = np.zeros((oh, ow))
    for i in range(oh):
        for j in range(ow):
            out[i, j] = (X[i:i+kh, j:j+kw] * K).sum()
    return out


rng = np.random.default_rng(42)

# Synthetic 8x8 "image" with a horizontal stripe
img = np.zeros((8, 8))
img[3:5, :] = 1.0

# Vertical-edge and horizontal-edge Sobel-style kernels
K_horiz = np.array([[ 1,  2,  1],
                     [ 0,  0,  0],
                     [-1, -2, -1]], dtype=float) / 8
K_vert  = np.array([[ 1,  0, -1],
                     [ 2,  0, -2],
                     [ 1,  0, -1]], dtype=float) / 8

fm_h = conv2d_manual(img, K_horiz)
fm_v = conv2d_manual(img, K_vert)

fig, axes = plt.subplots(1, 3, figsize=(11, 3.5))
for ax, data, title in zip(
        axes,
        [img, fm_h, fm_v],
        ["Input image (8x8)", "Horizontal-edge filter", "Vertical-edge filter"]):
    im = ax.imshow(data, cmap="RdBu_r", aspect="auto")
    ax.set_title(title, fontsize=11)
    plt.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
plt.suptitle("2-D convolution: same kernel, different spatial patterns detected")
plt.tight_layout()
plt.show()

print(f"Input shape  : {img.shape}")
print(f"Feature map  : {fm_h.shape}   (no padding, stride 1, kernel 3x3)")

# Verify with PyTorch
X_t = torch.tensor(img, dtype=torch.float32).unsqueeze(0).unsqueeze(0)  # (1,1,8,8)
K_t = torch.tensor(K_horiz, dtype=torch.float32).unsqueeze(0).unsqueeze(0)  # (1,1,3,3)
fm_torch = F.conv2d(X_t, K_t).squeeze().numpy()
print(f"Max abs diff manual vs PyTorch: {np.abs(fm_h - fm_torch).max():.2e}  [OK]")
No description has been provided for this image
Input shape  : (8, 8)
Feature map  : (6, 6)   (no padding, stride 1, kernel 3x3)
Max abs diff manual vs PyTorch: 0.00e+00  [OK]

Section 15.2 — Pooling, Strides, and Receptive Fields

Downsampling Spatial Dimensions

A convolution with $p=1$, $s=1$ preserves spatial size. After several such layers a $224\times224$ input would still produce $224\times224$ feature maps — computationally expensive and redundant. Two mechanisms reduce spatial resolution:

  • Stride $s > 1$: sample every $s$-th position during the convolution itself.
  • Max pooling: within each non-overlapping $k\times k$ window, keep only the maximum activation.

$$\text{MaxPool}(s{=}2): H_{\text{out}} = \lfloor H/2 \rfloor$$

Receptive Field

The receptive field of a neuron is the region of the original input that influences its value. Stacking two $3\times3$ conv layers gives a $5\times5$ effective receptive field with fewer parameters than a single $5\times5$ layer, and an extra non-linearity between them.

Depth vs. Width

As spatial dimensions shrink, the number of channels (independent filters at each layer) typically grows — from 3 (RGB) to 64, 128, 256, ... — allowing the network to represent increasingly abstract features.

$$\underbrace{224\times224\times3}_{\text{raw pixels}} \;\to\; 112\times112\times64 \;\to\; 56\times56\times128 \;\to\; \cdots \;\to\; \underbrace{1000}_{\text{classes}}$$

In [3]:
# ── Pooling and receptive-field illustration ──────────────────────────────
rng2 = np.random.default_rng(0)
feature_map = rng2.standard_normal((8, 8))

def maxpool2d(X: np.ndarray, k: int = 2) -> np.ndarray:
    h, w = X.shape[0] // k, X.shape[1] // k
    out = np.zeros((h, w))
    for i in range(h):
        for j in range(w):
            out[i, j] = X[i*k:(i+1)*k, j*k:(j+1)*k].max()
    return out

pooled = maxpool2d(feature_map, k=2)

fig, axes = plt.subplots(1, 2, figsize=(9, 3.5))
for ax, data, title in zip(
        axes,
        [feature_map, pooled],
        ["Feature map (8x8)", "After MaxPool 2x2 -> (4x4)"]):
    im = ax.imshow(data, cmap="viridis", aspect="auto")
    ax.set_title(title, fontsize=11)
    plt.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
plt.suptitle("MaxPool2d: spatial size halved, dominant activations preserved")
plt.tight_layout()
plt.show()

# Receptive field growth
print("Receptive field growth through stacked 3x3 convolutions (stride 1):")
rf = 1
for layer in range(1, 6):
    rf = rf + 2   # each 3x3 conv adds 1 on each side
    print(f"  After layer {layer}: receptive field = {rf}x{rf}")
No description has been provided for this image
Receptive field growth through stacked 3x3 convolutions (stride 1):
  After layer 1: receptive field = 3x3
  After layer 2: receptive field = 5x5
  After layer 3: receptive field = 7x7
  After layer 4: receptive field = 9x9
  After layer 5: receptive field = 11x11

Section 15.3 — Building a CNN with <code>nn.Module</code> — PyTorch Replaces NumPy

The Ch14 Pattern, Rewritten

In Chapter 14, one full training step was:

# NumPy (Ch14)
Y_hat, cache  = forward_pass(X, params, n_layers)
loss          = compute_loss(Y_hat, Y)
grads         = backward_pass(X, Y, params, cache, n_layers)
params        = {k: params[k] - lr * grads["d"+k] for k in params}

In PyTorch the same four steps become:

# PyTorch (Ch15)
out  = model(X)                # forward pass  — autograd records the graph
loss = criterion(out, y)       # compute loss
optimizer.zero_grad()          # clear gradients from last step
loss.backward()                # backprop      — autograd differentiates the graph
optimizer.step()               # weight update — optimizer reads .grad attributes

The algebra is identical. Autograd eliminates the manual backward_pass function; the GPU absorbs the matrix multiplications through .to(device).

Architecture: LeNet-style CNN

$$\text{Input}(1\times28\times28) \xrightarrow{\text{Conv 1x8, k=3}} \text{ReLU} \xrightarrow{\text{MaxPool}} 14\times14\times8 \xrightarrow{\text{Conv 8x16, k=3}} \text{ReLU} \xrightarrow{\text{MaxPool}} 7\times7\times16 \xrightarrow{\text{Flatten}} 784 \xrightarrow{\text{FC 128}} \text{ReLU} \xrightarrow{\text{FC 10}} \text{Softmax}$$

In [4]:
# ── LeNet-style CNN definition ────────────────────────────────────────────
class LeNetSmall(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 8, kernel_size=3, padding=1),   # -> (8, 28, 28)
            nn.ReLU(),
            nn.MaxPool2d(2),                              # -> (8, 14, 14)
            nn.Conv2d(8, 16, kernel_size=3, padding=1),  # -> (16, 14, 14)
            nn.ReLU(),
            nn.MaxPool2d(2),                              # -> (16, 7, 7)
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(16 * 7 * 7, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
        )

    def forward(self, x):
        return self.classifier(self.features(x))


model = LeNetSmall().to(device)
print(model)
print()

# Parameter count
n_params = sum(p.numel() for p in model.parameters())
dense_equiv = 784 * 128 + 128 * 10   # fully-connected baseline
print(f"CNN parameters   : {n_params:,}")
print(f"Dense equivalent : {dense_equiv:,}  (conv layers use far fewer weights)")

# Shape sanity check
dummy = torch.zeros(4, 1, 28, 28, device=device)
out   = model(dummy)
print(f"Output shape (batch=4, classes=10): {tuple(out.shape)}  [OK]")
LeNetSmall(
  (features): Sequential(
    (0): Conv2d(1, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU()
    (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(8, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (4): ReLU()
    (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Flatten(start_dim=1, end_dim=-1)
    (1): Linear(in_features=784, out_features=128, bias=True)
    (2): ReLU()
    (3): Linear(in_features=128, out_features=10, bias=True)
  )
)

CNN parameters   : 103,018
Dense equivalent : 101,632  (conv layers use far fewer weights)
Output shape (batch=4, classes=10): (4, 10)  [OK]

Section 15.4 — Training on MNIST: the Full PyTorch Loop

MNIST (LeCun et al. 1998) is 70,000 greyscale handwritten digit images ($28\times28$ pixels, 10 classes: 0–9). It remains the canonical first benchmark for image classifiers.

Data Pipeline

PyTorch's DataLoader handles batching, shuffling, and multi-process prefetch:

DataLoader(dataset, batch_size=128, shuffle=True, num_workers=0)

transforms.Normalize((0.1307,), (0.3081,)) centres and scales each pixel using the MNIST global mean and standard deviation — exactly the standardisation recommended in §14.6 for neural network inputs.

Training Loop

for epoch in range(N):
    for X_batch, y_batch in train_loader:
        out  = model(X_batch)
        loss = criterion(out, y_batch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

This is the template for every deep learning training loop you will encounter.

In [5]:
# ── Download MNIST and build DataLoaders ──────────────────────────────────
DATA_DIR = "../../data/mnist"

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,)),
])

train_ds = datasets.MNIST(DATA_DIR, train=True,  download=True, transform=transform)
test_ds  = datasets.MNIST(DATA_DIR, train=False, download=True, transform=transform)

train_loader = torch.utils.data.DataLoader(train_ds, batch_size=128, shuffle=True,  num_workers=0)
test_loader  = torch.utils.data.DataLoader(test_ds,  batch_size=256, shuffle=False, num_workers=0)

print(f"Training samples : {len(train_ds):,}")
print(f"Test samples     : {len(test_ds):,}")
print(f"Batches/epoch    : {len(train_loader)}")

# ── Training loop ─────────────────────────────────────────────────────────
model    = LeNetSmall().to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

N_EPOCHS = 5
history = {"train_loss": [], "test_acc": []}

for epoch in range(1, N_EPOCHS + 1):
    model.train()
    running_loss = 0.0
    for X_batch, y_batch in train_loader:
        X_batch, y_batch = X_batch.to(device), y_batch.to(device)
        optimizer.zero_grad()
        out  = model(X_batch)
        loss = criterion(out, y_batch)
        loss.backward()
        optimizer.step()
        running_loss += loss.item() * X_batch.size(0)

    avg_loss = running_loss / len(train_ds)
    history["train_loss"].append(avg_loss)

    # Test accuracy
    model.eval()
    correct = 0
    with torch.no_grad():
        for X_b, y_b in test_loader:
            X_b, y_b = X_b.to(device), y_b.to(device)
            preds = model(X_b).argmax(dim=1)
            correct += (preds == y_b).sum().item()
    acc = correct / len(test_ds)
    history["test_acc"].append(acc)
    print(f"  Epoch {epoch}/{N_EPOCHS}  loss={avg_loss:.4f}  test_acc={acc:.4f}")

# ── Visualise ─────────────────────────────────────────────────────────────
fig, axes = plt.subplots(1, 2, figsize=(11, 3.5))
axes[0].plot(range(1, N_EPOCHS + 1), history["train_loss"], marker="o", color="steelblue")
axes[0].set_title("Training loss")
axes[0].set_xlabel("Epoch")
axes[0].set_ylabel("Cross-entropy loss")

axes[1].plot(range(1, N_EPOCHS + 1), [a * 100 for a in history["test_acc"]],
             marker="s", color="darkorange")
axes[1].set_title("Test accuracy")
axes[1].set_xlabel("Epoch")
axes[1].set_ylabel("Accuracy (%)")
axes[1].set_ylim([95, 100])

plt.suptitle(f"LeNet-style CNN on MNIST  |  device: {device}")
plt.tight_layout()
plt.show()

# Final classification report (sample 1000 test examples)
model.eval()
all_preds, all_labels = [], []
with torch.no_grad():
    for X_b, y_b in test_loader:
        X_b = X_b.to(device)
        all_preds.extend(model(X_b).argmax(dim=1).cpu().numpy())
        all_labels.extend(y_b.numpy())
        if len(all_labels) >= 1000:
            break
print(classification_report(all_labels[:1000], all_preds[:1000], digits=3))
Training samples : 60,000
Test samples     : 10,000
Batches/epoch    : 469
  Epoch 1/5  loss=0.2858  test_acc=0.9740
  Epoch 2/5  loss=0.0783  test_acc=0.9794
  Epoch 3/5  loss=0.0552  test_acc=0.9844
  Epoch 4/5  loss=0.0424  test_acc=0.9872
  Epoch 5/5  loss=0.0354  test_acc=0.9875
No description has been provided for this image
              precision    recall  f1-score   support

           0      0.977     0.988     0.982        85
           1      1.000     0.992     0.996       126
           2      0.983     1.000     0.991       116
           3      0.981     0.991     0.986       107
           4      1.000     0.973     0.986       110
           5      0.989     0.989     0.989        87
           6      1.000     0.977     0.988        87
           7      0.990     0.990     0.990        99
           8      0.936     0.989     0.962        89
           9      0.989     0.957     0.973        94

    accuracy                          0.985      1000
   macro avg      0.984     0.985     0.984      1000
weighted avg      0.985     0.985     0.985      1000


Section 15.5 — Transfer Learning with ResNet-18

Why Pretrain?

Training a deep CNN from scratch on a small clinical dataset is rarely practical:

  • Modern architectures have millions of parameters and need millions of labelled images to learn general visual representations.
  • Labelled medical images are expensive: they require expert annotation.

Transfer learning solves this by starting from a model already trained on a large general dataset (ImageNet: 1.28 M images, 1000 classes) and fine-tuning the final layers on the new task.

ResNet-18 Architecture

ResNet (He et al. 2015) introduced skip connections (residual connections) that allow gradients to flow directly back through many layers, enabling networks 18–152 layers deep to train reliably:

$$\mathbf{a}^{[l+1]} = \text{ReLU}\!\left(\mathbf{z}^{[l+1]} + \mathbf{a}^{[l]}\right)$$

The addition $+\,\mathbf{a}^{[l]}$ is the skip. If the new layers learn nothing useful, their weights go to zero and the identity mapping is preserved — a form of implicit regularisation.

Fine-Tuning Recipe

  1. Load pretrained ResNet-18 weights (trained on ImageNet).
  2. Freeze all layers except the final fully-connected head.
  3. Replace the head with nn.Linear(512, n_classes).
  4. Train only the new head for a few epochs.

This requires far fewer labelled examples than training from scratch and converges in minutes, not hours.

In [6]:
# ── Transfer learning: fine-tune ResNet-18 on FashionMNIST ─────────────────
# FashionMNIST: same dimensions as MNIST (28x28, 10 classes, greyscale)
# but semantically harder (clothing items vs. handwritten digits).
# We use it here as a stand-in for a "new task" the pretrained model has not seen.

DATA_DIR = "../../data/mnist"

# ResNet-18 expects 3-channel, >=224x224 input; we upsample and convert to RGB
tl_transform = transforms.Compose([
    transforms.Resize(64),          # upsample 28->64 (faster than 224 for demo)
    transforms.Grayscale(num_output_channels=3),   # 1 channel -> 3 channels
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406],    # ImageNet mean
                         [0.229, 0.224, 0.225]),   # ImageNet std
])

fmnist_train = datasets.FashionMNIST(DATA_DIR, train=True,  download=True, transform=tl_transform)
fmnist_test  = datasets.FashionMNIST(DATA_DIR, train=False, download=True, transform=tl_transform)

# Use a 5000-sample subset for speed (mirrors a small clinical dataset)
subset_idx = torch.randperm(len(fmnist_train))[:5000]
fmnist_sub  = torch.utils.data.Subset(fmnist_train, subset_idx.tolist())

tl_train_loader = torch.utils.data.DataLoader(fmnist_sub,   batch_size=64, shuffle=True,  num_workers=0)
tl_test_loader  = torch.utils.data.DataLoader(fmnist_test,  batch_size=256, shuffle=False, num_workers=0)

FMNIST_CLASSES = ["T-shirt", "Trouser", "Pullover", "Dress", "Coat",
                  "Sandal",  "Shirt",   "Sneaker",  "Bag",   "Ankle boot"]

# ── Load ResNet-18 with pretrained ImageNet weights ────────────────────────
resnet = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)

# Freeze all layers
for param in resnet.parameters():
    param.requires_grad = False

# Replace the head: ResNet-18 final layer is Linear(512, 1000)
in_features = resnet.fc.in_features
resnet.fc   = nn.Linear(in_features, 10)   # 10 FashionMNIST classes
resnet      = resnet.to(device)

# Only head parameters will receive gradients
trainable = sum(p.numel() for p in resnet.parameters() if p.requires_grad)
total     = sum(p.numel() for p in resnet.parameters())
print(f"Trainable params : {trainable:,} / {total:,}  ({100*trainable/total:.1f}%)")

tl_criterion = nn.CrossEntropyLoss()
tl_optimizer = optim.Adam(resnet.fc.parameters(), lr=1e-3)

TL_EPOCHS = 4
tl_history = {"train_loss": [], "test_acc": []}

for epoch in range(1, TL_EPOCHS + 1):
    resnet.train()
    running_loss = 0.0
    for X_b, y_b in tl_train_loader:
        X_b, y_b = X_b.to(device), y_b.to(device)
        tl_optimizer.zero_grad()
        out  = resnet(X_b)
        loss = tl_criterion(out, y_b)
        loss.backward()
        tl_optimizer.step()
        running_loss += loss.item() * X_b.size(0)
    avg_loss = running_loss / len(fmnist_sub)
    tl_history["train_loss"].append(avg_loss)

    resnet.eval()
    correct = 0
    with torch.no_grad():
        for X_b, y_b in tl_test_loader:
            X_b, y_b = X_b.to(device), y_b.to(device)
            correct += (resnet(X_b).argmax(1) == y_b).sum().item()
    acc = correct / len(fmnist_test)
    tl_history["test_acc"].append(acc)
    print(f"  Epoch {epoch}/{TL_EPOCHS}  loss={avg_loss:.4f}  test_acc={acc:.4f}")

# ── Visualise ─────────────────────────────────────────────────────────────
fig, axes = plt.subplots(1, 2, figsize=(11, 3.5))
axes[0].plot(range(1, TL_EPOCHS + 1), tl_history["train_loss"],
             marker="o", color="steelblue")
axes[0].set_title("Transfer learning: training loss")
axes[0].set_xlabel("Epoch")
axes[0].set_ylabel("Cross-entropy loss")

axes[1].plot(range(1, TL_EPOCHS + 1), [a * 100 for a in tl_history["test_acc"]],
             marker="s", color="darkorange")
axes[1].set_title("Transfer learning: test accuracy")
axes[1].set_xlabel("Epoch")
axes[1].set_ylabel("Accuracy (%)")

plt.suptitle("ResNet-18 fine-tuned on FashionMNIST (5000-sample subset)")
plt.tight_layout()
plt.show()

# Classification report on first 1000 test examples
resnet.eval()
preds_tl, labels_tl = [], []
with torch.no_grad():
    for X_b, y_b in tl_test_loader:
        X_b = X_b.to(device)
        preds_tl.extend(resnet(X_b).argmax(1).cpu().numpy())
        labels_tl.extend(y_b.numpy())
        if len(labels_tl) >= 1000:
            break
print(classification_report(labels_tl[:1000], preds_tl[:1000],
                             target_names=FMNIST_CLASSES, digits=3))
Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /root/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth
  0%|          | 0.00/44.7M [00:00<?, ?B/s]
 23%|██▎       | 10.4M/44.7M [00:00<00:00, 108MB/s]
 49%|████▊     | 21.8M/44.7M [00:00<00:00, 114MB/s]
 75%|███████▍  | 33.4M/44.7M [00:00<00:00, 118MB/s]
100%|██████████| 44.7M/44.7M [00:00<00:00, 117MB/s]

Trainable params : 5,130 / 11,181,642  (0.0%)
  Epoch 1/4  loss=1.4266  test_acc=0.7168
  Epoch 2/4  loss=0.8468  test_acc=0.7438
  Epoch 3/4  loss=0.7227  test_acc=0.7557
  Epoch 4/4  loss=0.6510  test_acc=0.7690
No description has been provided for this image
              precision    recall  f1-score   support

     T-shirt      0.701     0.766     0.732       107
     Trouser      0.950     0.914     0.932       105
    Pullover      0.690     0.721     0.705       111
       Dress      0.646     0.667     0.656        93
        Coat      0.727     0.626     0.673       115
      Sandal      0.938     0.874     0.905        87
       Shirt      0.586     0.598     0.592        97
     Sneaker      0.802     0.895     0.846        95
         Bag      0.924     0.895     0.909        95
  Ankle boot      0.914     0.895     0.904        95

    accuracy                          0.781      1000
   macro avg      0.788     0.785     0.785      1000
weighted avg      0.784     0.781     0.782      1000


Section 15.6 — Summary & Quick Reference

Key Formulas

Concept Formula Where it appeared
2-D convolution $(X\star K)_{i,j}=\sum_{p,q}X_{i+p,j+q}\cdot K_{p,q}$ §15.1
Output spatial size $H_\text{out}=\lfloor(H+2p-k)/s\rfloor+1$ §15.1
MaxPool $\max$ over $k\times k$ non-overlapping window §15.2
Residual connection $\mathbf{a}^{[l+1]}=\text{ReLU}(\mathbf{z}^{[l+1]}+\mathbf{a}^{[l]})$ §15.5
PyTorch step loss.backward() $\to$ optimizer.step() §15.3–§15.4
Cross-entropy (multi-class) $\mathcal{L}=-\frac{1}{m}\sum_i\log\hat{p}_{y_i}$ §15.4, Ch13
Param sharing savings $k^2 C_{\text{in}} C_{\text{out}}$ vs. $HW C_{\text{in}}\cdot C_{\text{out}}$ §15.1

ML Connections

Concept Where it appears in ML
Parameter sharing The defining property of convolutional layers; enables translation equivariance
Autograd PyTorch's automatic differentiation — replaces manual backward_pass from Ch14
Adam optimiser Same algorithm as sklearn's MLPClassifier(solver='adam'); §15.4 and Ch11
Transfer learning Pretrain on large data; fine-tune on small clinical dataset — standard clinical AI workflow
Residual connections Enable very deep networks (18–152 layers) to train without vanishing gradients (Ch14 §14.2)
Softmax + cross-entropy Multi-class generalisation of sigmoid + BCE (Ch13, Ch14)

What's Next?

$$\boxed{\text{Ch15: CNNs and PyTorch} \;\longrightarrow\; \text{Attention Mechanisms and the Transformer Architecture}}$$

This chapter introduced spatial structure — how convolutional filters share weights across positions. The next chapter introduces sequential and global structure:

  • Self-attention: each token attends to every other token through learned query/key/value matrices — $\text{Attention}(Q,K,V)=\text{softmax}(QK^\top/\sqrt{d_k})V$.
  • The Transformer encoder: a stack of multi-head attention + feed-forward blocks that processes sequences in parallel (unlike RNNs).
  • Clinical relevance: BERT-style models encode clinical notes; Vision Transformers (ViT) replace convolutional feature extractors with patch embeddings.

All three inherit the same nn.Module pattern developed here.


References

  1. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. Introduced LeNet and the MNIST benchmark. https://doi.org/10.1109/5.726791

  2. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. CVPR 2016, 770–778. Introduced skip connections (ResNets); enables very deep networks. https://arxiv.org/abs/1512.03385

  3. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. NeurIPS 2012. AlexNet — launched the deep learning era in computer vision. https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html

  4. Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. ICLR 2015. VGG — demonstrated that depth with small (3x3) kernels outperforms large kernels. https://arxiv.org/abs/1409.1556

  5. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 9: Convolutional Networks. https://www.deeplearningbook.org/

  6. Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747. FashionMNIST dataset used in §15.5. https://arxiv.org/abs/1708.07747

  7. Litjens, G. et al. (2017). A survey on deep learning in medical image analysis. Medical Image Analysis, 42, 60–88. Comprehensive review of CNN applications in radiology, pathology, and ophthalmology. https://doi.org/10.1016/j.media.2017.07.005

  8. Esteva, A. et al. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542, 115–118. Landmark clinical application of transfer learning from ImageNet to dermoscopy images. https://doi.org/10.1038/nature21056

  9. Paszke, A. et al. (2019). PyTorch: An imperative style, high-performance deep learning library. NeurIPS 2019. Primary reference for the PyTorch framework. https://arxiv.org/abs/1912.01703

  10. Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimisation. ICLR 2015. Adam optimiser used in §15.4 and §15.5. https://arxiv.org/abs/1412.6980