Home / About Us / Volume I Contents / Ch 18: Vision Transformers

Statistical Basis of AI/ML

Vision Transformers and Multimodal Models

Chapter 18 -- Patch Embeddings, Image Classification, and Cross-Modal Representations

Chapter 17 showed how the Transformer encoder is industrialised for language via BERT. This chapter applies the identical self-attention mechanism to images -- splitting each image into a sequence of patches and treating pixel blocks exactly as word tokens.

Why This Chapter?

The Vision Transformer (ViT, Dosovitskiy et al., 2020) demonstrated that a pure Transformer encoder -- with no convolutions -- matches or beats convolutional networks on image classification when pre-trained at scale. Understanding ViT requires only the machinery already developed in Chapters 16-17.

Section Topic
§18.0 Setup and library overview
§18.1 From CNN to ViT: why patches work
§18.2 Patch embedding: converting images to token sequences
§18.3 MiniViT: building and training from scratch
§18.4 Attention map visualisation
§18.5 Multimodal models: CLIP and contrastive learning
§18.6 Medical imaging: BioViL, CheXpert, and MIMIC-CXR
§18.7 Summary and references

§18.0 Setup

Library Role in this chapter
torch / torch.nn MiniViT implementation, GPU training, AdamW, CosineAnnealingLR
torchvision Version verification (ViT built from scratch)
numpy Patch manipulation, CLIP similarity matrix
matplotlib / seaborn Training curves, attention maps, heatmaps
sklearn classification_report for per-class metrics

No model downloads are required in this chapter. The MiniViT is built from first principles using only torch.nn primitives.

In [1]:
# -- Imports ------------------------------------------------------------------
import warnings
warnings.filterwarnings("ignore")

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from sklearn.metrics import classification_report

sns.set_theme(style="whitegrid", font_scale=1.05)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

print(f"PyTorch     : {torch.__version__}")
print(f"torchvision : {torchvision.__version__}")
print(f"Device      : {device}")
if device.type == "cuda":
    print(f"GPU         : {torch.cuda.get_device_name(0)}")
    vram = torch.cuda.get_device_properties(0).total_memory / 1e9
    print(f"VRAM        : {vram:.1f} GB")
print("Chapter 18 -- Vision Transformers: environment ready.")
PyTorch     : 2.11.0+cu128
torchvision : 0.26.0+cu128
Device      : cuda
GPU         : NVIDIA GeForce RTX 5080
VRAM        : 17.1 GB
Chapter 18 -- Vision Transformers: environment ready.

§18.1 From CNN to ViT: Why Patches Work

The CNN inductive bias problem

Convolutional Neural Networks (Chapter 15) embed two strong inductive biases:

  1. Translation equivariance: the same filter detects a feature wherever it appears in the image.
  2. Locality: each feature depends only on a small receptive field.

These biases are useful when data is limited, but they limit what the model can learn: long-range dependencies require many stacked layers, and global context is only available at deep layers.

The ViT approach: treat image patches as tokens

ViT replaces convolutions with the following idea:

  1. Divide the image into non-overlapping $P \times P$ patches.
  2. Flatten each patch to a vector of length $P^2 C$ (where $C$ = channels).
  3. Project each flattened patch to the model dimension $d_{\text{model}}$ via a linear layer.
  4. Prepend a learnable [CLS] token (identical to BERT).
  5. Add learnable 1-D positional embeddings.
  6. Pass the sequence through standard Transformer encoder blocks.

The total number of patches (tokens) is:

$$N = \frac{H \times W}{P^2}$$

For a $32 \times 32$ image with $P = 4$: $N = 64$ patches.

Model Image size Patch size Tokens $N$ $d_{\text{model}}$ Params
ViT-B/16 224x224 16 196 768 86 M
ViT-L/16 224x224 16 196 1024 307 M
MiniViT (this chapter) 32x32 4 64 128 ~300 K

Self-attention is global from the first layer

Unlike a CNN, every patch token can attend to every other patch token at every layer. This means the model can capture long-range spatial dependencies without needing depth.

In [2]:
# -- Patch extraction demo on a synthetic 32x32 RGB image --------------------
torch.manual_seed(0)

IMG_SIZE  = 32
PATCH_SIZE = 4
N_PH = IMG_SIZE // PATCH_SIZE   # 8 patches per row
N_PW = IMG_SIZE // PATCH_SIZE   # 8 patches per col
N_PATCHES = N_PH * N_PW         # 64

# Synthetic quadrant-colored image (class 0: Red top-left)
img = torch.zeros(3, IMG_SIZE, IMG_SIZE)
h2, w2 = IMG_SIZE // 2, IMG_SIZE // 2
img[0, :h2, :w2] = 0.9   # red top-left
img[1, :h2, w2:] = 0.9   # green top-right
img[2, h2:, :w2] = 0.9   # blue bottom-left
img[0, h2:, w2:] = 0.9   # yellow bottom-right (red + green)
img[1, h2:, w2:] = 0.9

# Extract patches using torch.Tensor.unfold
# (C, H, W) -> (C, N_H, N_W, P, P) -> (N_H, N_W, C, P, P)
patches = (img.unfold(1, PATCH_SIZE, PATCH_SIZE)
              .unfold(2, PATCH_SIZE, PATCH_SIZE)
              .permute(1, 2, 0, 3, 4))   # (8, 8, 3, 4, 4)

# Visualise: original image + 4x4 sub-grid of patches
fig, axes = plt.subplots(1, 2, figsize=(10, 4))

axes[0].imshow(img.permute(1, 2, 0).clamp(0, 1).numpy())
axes[0].set_title("Original 32x32 image")
axes[0].axis("off")

GRID = 4
patch_canvas = np.zeros((GRID * PATCH_SIZE, GRID * PATCH_SIZE, 3))
for pi in range(GRID):
    for pj in range(GRID):
        patch_canvas[pi*PATCH_SIZE:(pi+1)*PATCH_SIZE,
                     pj*PATCH_SIZE:(pj+1)*PATCH_SIZE] = patches[pi, pj].permute(1, 2, 0).numpy()
axes[1].imshow(patch_canvas.clip(0, 1))
axes[1].set_title(f"Top-left {GRID}x{GRID} patches (patch_size={PATCH_SIZE})")
axes[1].axis("off")

plt.suptitle(f"Image ({IMG_SIZE}x{IMG_SIZE}, 3ch) "
             f"-> {N_PATCHES} patches of {PATCH_SIZE}x{PATCH_SIZE}x3 = "
             f"{PATCH_SIZE*PATCH_SIZE*3} features each", fontsize=11)
plt.tight_layout()
plt.show()

# Flat patch sequence
flat = patches.reshape(N_PATCHES, -1)   # (64, 48)
print(f"Patch sequence shape : {tuple(flat.shape)}")
print(f"  {N_PATCHES} tokens, each {PATCH_SIZE}x{PATCH_SIZE}x3 = {PATCH_SIZE*PATCH_SIZE*3} features")
print("[OK] patch extraction complete.")
No description has been provided for this image
Patch sequence shape : (64, 48)
  64 tokens, each 4x4x3 = 48 features
[OK] patch extraction complete.

§18.2 ViT Architecture: Modules and Data Flow

Input image (B, C, H, W)
        |
  PatchEmbedding
    - unfold H x W into N = (H*W/P^2) patches
    - Linear(P*P*C -> d_model)  [patch projection]
    - prepend learnable [CLS] token  -> (B, N+1, d_model)
    - add learnable positional embedding  (B, N+1, d_model)
        |
  EncoderBlock x n_blocks
    - LayerNorm -> MultiHeadAttention -> residual
    - LayerNorm -> FFN (GELU) -> residual
        |
  LayerNorm
        |
  [CLS] hidden state at position 0  (B, d_model)
        |
  Linear(d_model -> n_classes)
        |
  Cross-entropy loss

Key equations

Patch projection (learnable linear embedding): $$\mathbf{z}_0^{(i)} = \mathbf{E} \, \text{flatten}(\mathbf{x}^{(i)}) + \mathbf{e}_{\text{pos}}^{(i)}$$

where $\mathbf{E} \in \mathbb{R}^{d \times P^2 C}$ is the patch embedding matrix and $\mathbf{e}_{\text{pos}}$ is the learnable 1-D positional embedding.

Classification (after $L$ encoder blocks): $$\hat{y} = \text{Linear}(\text{LN}(\mathbf{z}_L^{[CLS]}))$$

Hyperparameters for MiniViT (this chapter)

Hyperparameter Value
Image size 32x32
Patch size $P$ 4
Tokens $N+1$ 65
$d_{\text{model}}$ 128
Attention heads 4
Encoder blocks 4
FFN expansion 2x
Dropout 0.10
Trainable params ~300 K
In [3]:
# -- MiniViT: patch embedding + encoder blocks + classification head ---------

class PatchEmbedding(nn.Module):
    def __init__(self, img_size, patch_size, in_channels, embed_dim):
        super().__init__()
        self.patch_size = patch_size
        n_patches = (img_size // patch_size) ** 2
        flat_dim  = patch_size * patch_size * in_channels
        self.proj      = nn.Linear(flat_dim, embed_dim)
        self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
        self.pos_embed = nn.Parameter(torch.zeros(1, n_patches + 1, embed_dim))
        nn.init.trunc_normal_(self.cls_token, std=0.02)
        nn.init.trunc_normal_(self.pos_embed, std=0.02)

    def forward(self, x):
        B, C, H, W = x.shape
        P = self.patch_size
        # unfold -> (B, C, N_H, N_W, P, P)
        x = x.unfold(2, P, P).unfold(3, P, P)
        # -> (B, N_H, N_W, C, P, P) -> (B, N, flat)
        _, _, N_H, N_W, _, _ = x.shape
        x = x.permute(0, 2, 3, 1, 4, 5).reshape(B, N_H * N_W, C * P * P)
        x = self.proj(x)                                # (B, N, embed_dim)
        cls = self.cls_token.expand(B, -1, -1)
        x   = torch.cat([cls, x], dim=1)               # (B, N+1, embed_dim)
        x   = x + self.pos_embed
        return x


class MHAttention(nn.Module):
    def __init__(self, embed_dim, n_heads):
        super().__init__()
        self.n_heads  = n_heads
        self.head_dim = embed_dim // n_heads
        self.scale    = self.head_dim ** -0.5
        self.qkv  = nn.Linear(embed_dim, embed_dim * 3)
        self.proj = nn.Linear(embed_dim, embed_dim)

    def forward(self, x, return_attn=False):
        B, N, D = x.shape
        H = self.n_heads
        qkv  = self.qkv(x).reshape(B, N, 3, H, self.head_dim).permute(2, 0, 3, 1, 4)
        q, k, v = qkv[0], qkv[1], qkv[2]
        attn = (q @ k.transpose(-2, -1)) * self.scale
        attn = attn.softmax(dim=-1)
        out  = (attn @ v).transpose(1, 2).reshape(B, N, D)
        out  = self.proj(out)
        if return_attn:
            return out, attn
        return out


class EncoderBlock(nn.Module):
    def __init__(self, embed_dim, n_heads, mlp_ratio=2, drop=0.10):
        super().__init__()
        self.norm1 = nn.LayerNorm(embed_dim)
        self.attn  = MHAttention(embed_dim, n_heads)
        self.norm2 = nn.LayerNorm(embed_dim)
        mlp_dim    = embed_dim * mlp_ratio
        self.mlp   = nn.Sequential(
            nn.Linear(embed_dim, mlp_dim), nn.GELU(),
            nn.Dropout(drop),
            nn.Linear(mlp_dim, embed_dim), nn.Dropout(drop),
        )

    def forward(self, x, return_attn=False):
        if return_attn:
            attn_out, attn_w = self.attn(self.norm1(x), return_attn=True)
            x = x + attn_out
            x = x + self.mlp(self.norm2(x))
            return x, attn_w
        x = x + self.attn(self.norm1(x))
        x = x + self.mlp(self.norm2(x))
        return x


class MiniViT(nn.Module):
    def __init__(self, img_size=32, patch_size=4, in_channels=3,
                 embed_dim=128, n_heads=4, n_blocks=4, n_classes=4, mlp_ratio=2):
        super().__init__()
        self.patch_embed = PatchEmbedding(img_size, patch_size, in_channels, embed_dim)
        self.blocks = nn.ModuleList([
            EncoderBlock(embed_dim, n_heads, mlp_ratio) for _ in range(n_blocks)
        ])
        self.norm = nn.LayerNorm(embed_dim)
        self.head = nn.Linear(embed_dim, n_classes)

    def forward(self, x, return_attn=False):
        x = self.patch_embed(x)
        attn_weights = []
        for blk in self.blocks:
            if return_attn:
                x, aw = blk(x, return_attn=True)
                attn_weights.append(aw)
            else:
                x = blk(x)
        x      = self.norm(x)
        logits = self.head(x[:, 0])
        if return_attn:
            return logits, attn_weights
        return logits


vit = MiniViT().to(device)
n_params = sum(p.numel() for p in vit.parameters() if p.requires_grad)
print(f"MiniViT trainable params : {n_params:,}")

# Verify forward pass
dummy  = torch.randn(2, 3, IMG_SIZE, IMG_SIZE).to(device)
logits = vit(dummy)
print(f"Input shape  : {tuple(dummy.shape)}")
print(f"Output shape : {tuple(logits.shape)}")

# Verify attention return path
logits2, attn_list = vit(dummy, return_attn=True)
print(f"Attention list length       : {len(attn_list)} blocks")
print(f"Last block attn shape       : {tuple(attn_list[-1].shape)}")
# Expected: (B, n_heads, N+1, N+1) = (2, 4, 65, 65)
print("[OK] MiniViT architecture built.")
MiniViT trainable params : 545,412
Input shape  : (2, 3, 32, 32)
Output shape : (2, 4)
Attention list length       : 4 blocks
Last block attn shape       : (2, 4, 65, 65)
[OK] MiniViT architecture built.

§18.3 Training MiniViT on Synthetic Quadrant Images

To isolate the ViT mechanism, we use a synthetic task designed to require spatial attention:

  • 4 classes (one per image quadrant):
    • Class 0 -- Red patch in the top-left quadrant
    • Class 1 -- Green patch in the top-right quadrant
    • Class 2 -- Blue patch in the bottom-left quadrant
    • Class 3 -- Yellow patch in the bottom-right quadrant
  • Each image is 32x32 pixels. The coloured quadrant occupies exactly 16x16 pixels (4 patches x 4 patches). The remaining three quadrants are black + Gaussian noise.
  • Why this task? The class-discriminating signal is perfectly spatially localised to one quadrant. A well-trained ViT should concentrate its [CLS] attention on the relevant quadrant -- which we verify in §18.4.

Training configuration

Setting Value
Train samples 800 (200 per class)
Test samples 200 (50 per class)
Optimiser AdamW, lr=1e-3, weight_decay=0.05
LR schedule CosineAnnealingLR, T_max=10
Epochs 10
Batch size 64
In [4]:
# -- Synthetic QuadrantDataset + training loop --------------------------------
import random
from torch.utils.data import Dataset, DataLoader

random.seed(42)
torch.manual_seed(42)
rng = np.random.default_rng(42)

NOISE_STD = 0.15

def make_quadrant_image(cls_idx):
    h2, w2 = IMG_SIZE // 2, IMG_SIZE // 2
    img = torch.zeros(3, IMG_SIZE, IMG_SIZE)
    if cls_idx == 0:      # red top-left
        img[0, :h2, :w2] = 1.0
    elif cls_idx == 1:    # green top-right
        img[1, :h2, w2:] = 1.0
    elif cls_idx == 2:    # blue bottom-left
        img[2, h2:, :w2] = 1.0
    else:                 # yellow (red+green) bottom-right
        img[0, h2:, w2:] = 1.0
        img[1, h2:, w2:] = 1.0
    return (img + torch.randn_like(img) * NOISE_STD).clamp(0.0, 1.0)


class QuadrantDataset(Dataset):
    def __init__(self, n_per_class):
        self.items = []
        for cls in range(4):
            for _ in range(n_per_class):
                self.items.append((make_quadrant_image(cls), cls))
        random.shuffle(self.items)

    def __len__(self):
        return len(self.items)

    def __getitem__(self, i):
        return self.items[i]


tr_ds = QuadrantDataset(200)   # 800 train
te_ds = QuadrantDataset(50)    # 200 test
tr_ld = DataLoader(tr_ds, batch_size=64, shuffle=True)
te_ld = DataLoader(te_ds, batch_size=128)

# Fresh MiniViT for training
vit2 = MiniViT().to(device)
opt  = torch.optim.AdamW(vit2.parameters(), lr=1e-3, weight_decay=0.05)
sch  = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=10)

EPOCHS = 10
tr_losses, te_accs = [], []

for epoch in range(1, EPOCHS + 1):
    vit2.train()
    ep_loss = 0.0
    for imgs, labs in tr_ld:
        imgs, labs = imgs.to(device), labs.to(device)
        loss = F.cross_entropy(vit2(imgs), labs)
        opt.zero_grad()
        loss.backward()
        opt.step()
        ep_loss += loss.item() * imgs.size(0)
    sch.step()
    avg_loss = ep_loss / len(tr_ds)

    vit2.eval()
    preds, gts = [], []
    with torch.no_grad():
        for imgs, labs in te_ld:
            preds.extend(vit2(imgs.to(device)).argmax(1).cpu().tolist())
            gts.extend(labs.tolist())
    acc = sum(p == g for p, g in zip(preds, gts)) / len(gts)
    tr_losses.append(avg_loss)
    te_accs.append(acc)
    print(f"  Epoch {epoch:2d}/{EPOCHS}  loss={avg_loss:.4f}  test_acc={acc:.4f}")

# Training curves
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3.5))
ax1.plot(range(1, EPOCHS + 1), tr_losses, "o-", color="steelblue", lw=2)
ax1.set_title("Training Loss"); ax1.set_xlabel("Epoch"); ax1.set_ylabel("Cross-entropy")
ax2.plot(range(1, EPOCHS + 1), [a * 100 for a in te_accs], "o-", color="darkorange", lw=2)
ax2.axhline(25, color="grey", ls="--", label="Chance (25%)")
ax2.set_title("Test Accuracy (%)"); ax2.set_xlabel("Epoch"); ax2.legend()
plt.tight_layout()
plt.show()

CLASS_NAMES = ["Red-TL", "Green-TR", "Blue-BL", "Yellow-BR"]
print(classification_report(gts, preds, target_names=CLASS_NAMES))
  Epoch  1/10  loss=0.6056  test_acc=1.0000
  Epoch  2/10  loss=0.0267  test_acc=1.0000
  Epoch  3/10  loss=0.0094  test_acc=1.0000
  Epoch  4/10  loss=0.0063  test_acc=1.0000
  Epoch  5/10  loss=0.0048  test_acc=1.0000
  Epoch  6/10  loss=0.0041  test_acc=1.0000
  Epoch  7/10  loss=0.0037  test_acc=1.0000
  Epoch  8/10  loss=0.0035  test_acc=1.0000
  Epoch  9/10  loss=0.0034  test_acc=1.0000
  Epoch 10/10  loss=0.0034  test_acc=1.0000
No description has been provided for this image
              precision    recall  f1-score   support

      Red-TL       1.00      1.00      1.00        50
    Green-TR       1.00      1.00      1.00        50
     Blue-BL       1.00      1.00      1.00        50
   Yellow-BR       1.00      1.00      1.00        50

    accuracy                           1.00       200
   macro avg       1.00      1.00      1.00       200
weighted avg       1.00      1.00      1.00       200

§18.4 Attention Map Visualisation

After training, we can inspect which patches the model attends to when classifying an image. The [CLS] token aggregates information from all patch tokens; its attention weights reveal the model's spatial focus.

Extracting [CLS] attention

From the last encoder block, the attention tensor has shape $(B, H, N+1, N+1)$ where $H$ is the number of heads and $N+1$ = 65 (1 [CLS]

  • 64 patches).
  1. Slice row 0 (the [CLS] token) attending to columns 1: (patch tokens): $\mathbf{a} \in \mathbb{R}^{H \times N}$.
  2. Average over heads: $\bar{\mathbf{a}} \in \mathbb{R}^{N}$.
  3. Reshape to the patch grid $(N_H, N_W) = (8, 8)$.
  4. Upsample to image resolution $(32, 32)$ via nearest-neighbour repetition.
  5. Overlay as a heatmap on the original image.

A correctly trained ViT should show high attention concentrated in the coloured quadrant and near-zero attention elsewhere.

In [5]:
# -- [CLS] attention map visualisation (last encoder block, head-averaged) ---
vit2.eval()

fig, axes = plt.subplots(2, 4, figsize=(13, 6.5))

for cls_idx in range(4):
    # Find one image of this class from the test set
    test_img = None
    for img_t, lab_t in te_ds.items:
        if lab_t == cls_idx:
            test_img = img_t
            break

    inp = test_img.unsqueeze(0).to(device)
    with torch.no_grad():
        _, attn_list = vit2(inp, return_attn=True)

    # Last block: (1, n_heads, N+1, N+1)
    attn_last = attn_list[-1][0]          # (n_heads, N+1, N+1)
    cls_attn  = attn_last[:, 0, 1:]       # (n_heads, N_patches)
    cls_attn  = cls_attn.mean(0)          # (N_patches,) averaged over heads
    cls_attn  = cls_attn.reshape(N_PH, N_PW).cpu().numpy()   # (8, 8)

    # Upsample to image size with nearest-neighbour (kron)
    attn_map  = np.kron(cls_attn, np.ones((PATCH_SIZE, PATCH_SIZE)))   # (32, 32)
    lo, hi    = attn_map.min(), attn_map.max()
    attn_map  = (attn_map - lo) / (hi - lo + 1e-8)

    ax_img  = axes[0, cls_idx]
    ax_attn = axes[1, cls_idx]

    ax_img.imshow(test_img.permute(1, 2, 0).clamp(0, 1).numpy())
    ax_img.set_title(CLASS_NAMES[cls_idx], fontsize=11)
    ax_img.axis("off")

    ax_attn.imshow(test_img.permute(1, 2, 0).clamp(0, 1).numpy())
    ax_attn.imshow(attn_map, cmap="hot", alpha=0.60)
    ax_attn.set_title("[CLS] attention", fontsize=11)
    ax_attn.axis("off")

axes[0, 0].set_ylabel("Image", fontsize=11)
axes[1, 0].set_ylabel("[CLS] Attn", fontsize=11)
plt.suptitle(
    "MiniViT [CLS] Attention Maps -- Last Encoder Block (averaged over 4 heads)\n"
    "Bright = high attention; the model focuses on the discriminative quadrant.",
    fontsize=11,
)
plt.tight_layout()
plt.show()
print("[OK] attention map visualisation complete.")
No description has been provided for this image
[OK] attention map visualisation complete.

§18.5 Multimodal Models: CLIP and Contrastive Learning

What is CLIP?

CLIP (Contrastive Language-Image Pre-Training; Radford et al., 2021) trains two encoders jointly:

  • An image encoder (ResNet or ViT) that maps images to a shared embedding space.
  • A text encoder (Transformer) that maps natural-language captions to the same space.

The training objective is InfoNCE contrastive loss: given a batch of $N$ (image, text) pairs, maximise the cosine similarity of matching pairs and minimise it for the $N^2 - N$ mismatched pairs:

$$\mathcal{L}_{\text{CLIP}} = -\frac{1}{2N} \sum_{i=1}^{N} \left[ \log \frac{e^{\text{sim}(v_i, t_i) / \tau}}{\sum_j e^{\text{sim}(v_i, t_j) / \tau}} + \log \frac{e^{\text{sim}(t_i, v_i) / \tau}}{\sum_j e^{\text{sim}(t_j, v_i) / \tau}} \right]$$

where $\text{sim}(\cdot, \cdot)$ is cosine similarity and $\tau$ is a learnable temperature.

Key property: zero-shot classification

After training on 400 M image-text pairs (LAION-5B for open-CLIP), the model can classify any image into any set of classes by:

  1. Encoding class names as text: "a photo of a {class}".
  2. Encoding the query image.
  3. Returning the class whose text embedding has the highest cosine similarity.

No fine-tuning is required.

Contrastive similarity matrix

The ideal training signal is a $N \times N$ matrix where the diagonal equals 1 (matched pair) and off-diagonal elements approach 0 (mismatched pairs).

In [6]:
# -- Synthetic CLIP-style cosine similarity matrix (no model download) -------
rng = np.random.default_rng(7)

CLIP_CLASSES = [
    "No nodule",
    "Lung opacity",
    "Cardiomegaly",
    "Pleural effusion",
    "Pneumothorax",
]
N_CLS = len(CLIP_CLASSES)
DIM   = 64   # embedding dimension

# Simulate matched (image, text) embeddings:
# same-class pairs share a common base direction; noise breaks identity.
img_embs, txt_embs = [], []
for _ in range(N_CLS):
    center = rng.standard_normal(DIM)
    center /= np.linalg.norm(center)
    img_embs.append(center + rng.standard_normal(DIM) * 0.05)
    txt_embs.append(center + rng.standard_normal(DIM) * 0.05)

img_embs = np.stack(img_embs)
txt_embs = np.stack(txt_embs)

# L2 normalise both
img_embs /= np.linalg.norm(img_embs, axis=1, keepdims=True)
txt_embs /= np.linalg.norm(txt_embs, axis=1, keepdims=True)

sim_mat = img_embs @ txt_embs.T   # (N_CLS, N_CLS)

fig, ax = plt.subplots(figsize=(6.5, 5.5))
sns.heatmap(
    sim_mat, annot=True, fmt=".2f", cmap="coolwarm", center=0,
    xticklabels=CLIP_CLASSES, yticklabels=CLIP_CLASSES,
    linewidths=0.5, vmin=-1.0, vmax=1.0, ax=ax,
)
ax.set_title(
    "Synthetic CLIP-style Image-Text Cosine Similarity\n"
    "(diagonal = matched pairs; off-diagonal = mismatched)",
    fontsize=10,
)
ax.set_xlabel("Text embeddings")
ax.set_ylabel("Image embeddings")
plt.xticks(rotation=30, ha="right")
plt.tight_layout()
plt.show()

diag_mean    = np.diag(sim_mat).mean()
n_offdiag    = N_CLS * (N_CLS - 1)
offdiag_mean = (sim_mat.sum() - np.trace(sim_mat)) / n_offdiag
print(f"Diagonal mean (matched pairs)    : {diag_mean:.4f}")
print(f"Off-diagonal mean (mismatched)   : {offdiag_mean:.4f}")
print(f"Contrast ratio (diag / offdiag)  : {diag_mean / max(abs(offdiag_mean), 1e-6):.1f}x")
print("[OK] CLIP similarity demo complete.")
No description has been provided for this image
Diagonal mean (matched pairs)    : 0.8852
Off-diagonal mean (mismatched)   : -0.0248
Contrast ratio (diag / offdiag)  : 35.7x
[OK] CLIP similarity demo complete.

§18.6 Medical Imaging: BioViL, CheXpert, and MIMIC-CXR

Why ViT fits medical imaging

Medical images (X-rays, MRI, histopathology slides) share a key property with natural images: spatially localised pathology. A ViT's global self-attention from the first layer is particularly well-suited to detecting:

  • Long-range correlations between anatomical structures.
  • Subtle texture differences across large field-of-view regions.
  • Relationships between lesion location and surrounding tissue context.

Key datasets

Dataset Size Modality Labels
CheXpert (Irvin et al., 2019) 224 K Chest X-ray 14 pathology labels (multi-label)
MIMIC-CXR (Johnson et al., 2019) 227 K Chest X-ray Radiology reports + structured labels
PatchCamelyon (Veeling et al., 2018) 327 K Histopathology Tumour vs normal (binary)

Domain-adapted vision-language models

BioViL (Bannur et al., 2023): A CLIP-style model trained on chest X-ray / report pairs from MIMIC-CXR.

  • Image encoder: ResNet-50 fine-tuned on CheXpert.
  • Text encoder: ClinicalBERT (Chapter 17) fine-tuned on radiology reports.
  • Contrastive objective: InfoNCE on matched X-ray / impression pairs.
  • Zero-shot classification: encode pathology names as text, rank by cosine similarity.

rad-DINO (Perez-Garcia et al., 2024): A DINO self-supervised ViT trained exclusively on chest X-rays. No text labels required -- learns spatial representations through self-distillation between patch views.

Practical fine-tuning pipeline

Pre-trained ViT (ImageNet)
        |
Continue pre-training on CheXpert (domain shift)
        |
Fine-tune on task: multi-label classification / report grounding
        |
Evaluate on held-out MIMIC-CXR split

The key insight from Chapter 17 transfers directly: pre-train once on large unlabelled corpora, then fine-tune on small annotated datasets. The only change is that the tokeniser operates on image patches instead of word pieces.

§18.7 Summary

Key Concepts

Concept Description
Patch tokenisation Divide image into $P \times P$ non-overlapping patches; flatten and project to $d_{\text{model}}$
Learnable positional embedding 1-D sequence of vectors added to patch tokens; learned jointly with model weights
[CLS] token Prepended learnable token whose final hidden state represents the whole image
Global attention Every patch attends to every other patch at every layer; no locality bias
ViT vs CNN CNN: translation equivariance, local receptive field, data-efficient. ViT: global context, scale-hungry, benefits from pre-training
CLIP Jointly trained image + text encoders with InfoNCE contrastive loss; enables zero-shot classification
BioViL / rad-DINO Domain-adapted ViTs for chest X-ray; same architecture, different pre-training corpus

Connections to Earlier Chapters

Chapter Connection
Ch14 (Neural Networks) Cross-entropy loss, AdamW, batch training loop
Ch15 (CNNs) Transfer learning paradigm; ViT replaces convolutions with self-attention
Ch16 (Transformers) Encoder blocks, multi-head attention, positional encoding -- identical in ViT
Ch17 (BERT) [CLS] token, pre-train/fine-tune paradigm, HuggingFace pipeline

What's Next?

Chapter 19 explores Graph Neural Networks (GNNs) -- a third structural inductive bias (after locality in CNNs and sequence order in Transformers) that models data as irregular graphs. Applications include molecular property prediction, protein structure reasoning, and knowledge graph completion.


Section 18.8 — Summary & Quick Reference

Key Formulas

Concept Formula Python
Patch count $N = (H/P)^2$ for square images (e.g., $224^2/16^2 = 196$) n = (img_size // patch_size) ** 2
Patch projection $\mathbf{e}_i = \mathbf{x}_i^{\text{patch}}\mathbf{W}_E + \mathbf{p}_i$ nn.Linear(patch_dim, d_model)
Self-attention (Ch16) $\text{Attn} = \text{softmax}(\mathbf{Q}\mathbf{K}^T/\sqrt{d_k})\mathbf{V}$ nn.MultiheadAttention(d_model, n_heads)
CLIP InfoNCE loss $\mathcal{L} = -\tfrac{1}{N}\sum_i \log \dfrac{\exp(\mathbf{v}_i\cdot\mathbf{t}_i/\tau)}{\sum_j \exp(\mathbf{v}_i\cdot\mathbf{t}_j/\tau)}$ F.cross_entropy(logits_vi, targets)
Cosine similarity (Ch10) $\cos\theta = \mathbf{a}\cdot\mathbf{b}/(\|\mathbf{a}\|\,\|\mathbf{b}\|)$ F.cosine_similarity(img_emb, txt_emb)

ML Connections

Concept Where It Appears
Patch as token Images treated as sequences; same Transformer encoder as Ch16 and Ch17
[CLS] token Global image representation; identical role to BERT's [CLS] in Ch17
Pre-train / fine-tune Same paradigm as BERT (Ch17) and CNNs (Ch15); scale then specialise
Contrastive learning (CLIP) InfoNCE aligns image and text embedding spaces using cosine similarity (Ch10)
BioViL / rad-DINO Domain-adapted ViTs for chest X-ray; pairs ClinicalBERT (Ch17) with patch attention

What's Next?

$$\boxed{\text{Ch 18: Vision Transformers}} \;\to\; \boxed{\text{Ch 19: Graph Neural Networks}} \;\to\; \boxed{\text{Ch 20: Cognitive Agent}} \;\to\; \boxed{\text{Ch 21: Bridge to Vol II}}$$

Chapter 19 introduces the third structural inductive bias in deep learning: the graph. Where CNNs assume a grid and Transformers assume a sequence, Graph Neural Networks operate on irregular topology -- atoms and bonds in molecules, residues in proteins, concepts in medical knowledge graphs. Self-attention (Ch16) is revealed to be a fully-connected GNN with learned edge weights.

References

  1. Dosovitskiy A, Beyer L, Kolesnikov A, et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021.
  2. Radford A, Kim JW, Hallacy C, et al. (2021). Learning transferable visual models from natural language supervision (CLIP). ICML, 139, 8748-8763.
  3. Bannur S, Hyland SL, Liu Q, et al. (2023). Learning to exploit temporal structure for biomedical vision-language processing (BioViL). CVPR.
  4. Perez-Garcia F, Sharma H, Bond-Taylor S, et al. (2024). A foundation model for clinical-grade computational pathology and rare cancers (rad-DINO). arXiv:2401.10815.
  5. Irvin J, Rajpurkar P, Ko M, et al. (2019). CheXpert: a large chest radiograph dataset with uncertainty labels. AAAI, 590-597.
  6. Johnson AEW, Pollard TJ, Berkowitz SJ, et al. (2019). MIMIC-CXR: a large publicly available database of labeled chest radiographs. arXiv:1901.07042.
  7. Touvron H, Cord M, Douze M, et al. (2021). Training data-efficient image transformers and distillation through attention (DeiT). ICML.
  8. Caron M, Touvron H, Misra I, et al. (2021). Emerging properties in self-supervised vision Transformers (DINO). ICCV, 9650-9660.
  9. He K, Chen X, Xie S, et al. (2022). Masked autoencoders are scalable vision learners (MAE). CVPR.
  10. Veeling BS, Linmans J, Winkens J, et al. (2018). Rotation equivariant CNNs for digital pathology (PatchCamelyon). MICCAI.