Statistical Basis of AI/ML¶
Vision Transformers and Multimodal Models¶
Chapter 18 -- Patch Embeddings, Image Classification, and Cross-Modal Representations¶
Chapter 17 showed how the Transformer encoder is industrialised for language via BERT. This chapter applies the identical self-attention mechanism to images -- splitting each image into a sequence of patches and treating pixel blocks exactly as word tokens.
Why This Chapter?¶
The Vision Transformer (ViT, Dosovitskiy et al., 2020) demonstrated that a pure Transformer encoder -- with no convolutions -- matches or beats convolutional networks on image classification when pre-trained at scale. Understanding ViT requires only the machinery already developed in Chapters 16-17.
| Section | Topic |
|---|---|
| §18.0 | Setup and library overview |
| §18.1 | From CNN to ViT: why patches work |
| §18.2 | Patch embedding: converting images to token sequences |
| §18.3 | MiniViT: building and training from scratch |
| §18.4 | Attention map visualisation |
| §18.5 | Multimodal models: CLIP and contrastive learning |
| §18.6 | Medical imaging: BioViL, CheXpert, and MIMIC-CXR |
| §18.7 | Summary and references |
§18.0 Setup¶
| Library | Role in this chapter |
|---|---|
torch / torch.nn |
MiniViT implementation, GPU training, AdamW, CosineAnnealingLR |
torchvision |
Version verification (ViT built from scratch) |
numpy |
Patch manipulation, CLIP similarity matrix |
matplotlib / seaborn |
Training curves, attention maps, heatmaps |
sklearn |
classification_report for per-class metrics |
No model downloads are required in this chapter. The MiniViT is built from
first principles using only torch.nn primitives.
# -- Imports ------------------------------------------------------------------
import warnings
warnings.filterwarnings("ignore")
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from sklearn.metrics import classification_report
sns.set_theme(style="whitegrid", font_scale=1.05)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"PyTorch : {torch.__version__}")
print(f"torchvision : {torchvision.__version__}")
print(f"Device : {device}")
if device.type == "cuda":
print(f"GPU : {torch.cuda.get_device_name(0)}")
vram = torch.cuda.get_device_properties(0).total_memory / 1e9
print(f"VRAM : {vram:.1f} GB")
print("Chapter 18 -- Vision Transformers: environment ready.")
§18.1 From CNN to ViT: Why Patches Work¶
The CNN inductive bias problem¶
Convolutional Neural Networks (Chapter 15) embed two strong inductive biases:
- Translation equivariance: the same filter detects a feature wherever it appears in the image.
- Locality: each feature depends only on a small receptive field.
These biases are useful when data is limited, but they limit what the model can learn: long-range dependencies require many stacked layers, and global context is only available at deep layers.
The ViT approach: treat image patches as tokens¶
ViT replaces convolutions with the following idea:
- Divide the image into non-overlapping $P \times P$ patches.
- Flatten each patch to a vector of length $P^2 C$ (where $C$ = channels).
- Project each flattened patch to the model dimension $d_{\text{model}}$ via a linear layer.
- Prepend a learnable
[CLS]token (identical to BERT). - Add learnable 1-D positional embeddings.
- Pass the sequence through standard Transformer encoder blocks.
The total number of patches (tokens) is:
$$N = \frac{H \times W}{P^2}$$
For a $32 \times 32$ image with $P = 4$: $N = 64$ patches.
| Model | Image size | Patch size | Tokens $N$ | $d_{\text{model}}$ | Params |
|---|---|---|---|---|---|
| ViT-B/16 | 224x224 | 16 | 196 | 768 | 86 M |
| ViT-L/16 | 224x224 | 16 | 196 | 1024 | 307 M |
| MiniViT (this chapter) | 32x32 | 4 | 64 | 128 | ~300 K |
Self-attention is global from the first layer¶
Unlike a CNN, every patch token can attend to every other patch token at every layer. This means the model can capture long-range spatial dependencies without needing depth.
# -- Patch extraction demo on a synthetic 32x32 RGB image --------------------
torch.manual_seed(0)
IMG_SIZE = 32
PATCH_SIZE = 4
N_PH = IMG_SIZE // PATCH_SIZE # 8 patches per row
N_PW = IMG_SIZE // PATCH_SIZE # 8 patches per col
N_PATCHES = N_PH * N_PW # 64
# Synthetic quadrant-colored image (class 0: Red top-left)
img = torch.zeros(3, IMG_SIZE, IMG_SIZE)
h2, w2 = IMG_SIZE // 2, IMG_SIZE // 2
img[0, :h2, :w2] = 0.9 # red top-left
img[1, :h2, w2:] = 0.9 # green top-right
img[2, h2:, :w2] = 0.9 # blue bottom-left
img[0, h2:, w2:] = 0.9 # yellow bottom-right (red + green)
img[1, h2:, w2:] = 0.9
# Extract patches using torch.Tensor.unfold
# (C, H, W) -> (C, N_H, N_W, P, P) -> (N_H, N_W, C, P, P)
patches = (img.unfold(1, PATCH_SIZE, PATCH_SIZE)
.unfold(2, PATCH_SIZE, PATCH_SIZE)
.permute(1, 2, 0, 3, 4)) # (8, 8, 3, 4, 4)
# Visualise: original image + 4x4 sub-grid of patches
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].imshow(img.permute(1, 2, 0).clamp(0, 1).numpy())
axes[0].set_title("Original 32x32 image")
axes[0].axis("off")
GRID = 4
patch_canvas = np.zeros((GRID * PATCH_SIZE, GRID * PATCH_SIZE, 3))
for pi in range(GRID):
for pj in range(GRID):
patch_canvas[pi*PATCH_SIZE:(pi+1)*PATCH_SIZE,
pj*PATCH_SIZE:(pj+1)*PATCH_SIZE] = patches[pi, pj].permute(1, 2, 0).numpy()
axes[1].imshow(patch_canvas.clip(0, 1))
axes[1].set_title(f"Top-left {GRID}x{GRID} patches (patch_size={PATCH_SIZE})")
axes[1].axis("off")
plt.suptitle(f"Image ({IMG_SIZE}x{IMG_SIZE}, 3ch) "
f"-> {N_PATCHES} patches of {PATCH_SIZE}x{PATCH_SIZE}x3 = "
f"{PATCH_SIZE*PATCH_SIZE*3} features each", fontsize=11)
plt.tight_layout()
plt.show()
# Flat patch sequence
flat = patches.reshape(N_PATCHES, -1) # (64, 48)
print(f"Patch sequence shape : {tuple(flat.shape)}")
print(f" {N_PATCHES} tokens, each {PATCH_SIZE}x{PATCH_SIZE}x3 = {PATCH_SIZE*PATCH_SIZE*3} features")
print("[OK] patch extraction complete.")
§18.2 ViT Architecture: Modules and Data Flow¶
Input image (B, C, H, W)
|
PatchEmbedding
- unfold H x W into N = (H*W/P^2) patches
- Linear(P*P*C -> d_model) [patch projection]
- prepend learnable [CLS] token -> (B, N+1, d_model)
- add learnable positional embedding (B, N+1, d_model)
|
EncoderBlock x n_blocks
- LayerNorm -> MultiHeadAttention -> residual
- LayerNorm -> FFN (GELU) -> residual
|
LayerNorm
|
[CLS] hidden state at position 0 (B, d_model)
|
Linear(d_model -> n_classes)
|
Cross-entropy loss
Key equations¶
Patch projection (learnable linear embedding): $$\mathbf{z}_0^{(i)} = \mathbf{E} \, \text{flatten}(\mathbf{x}^{(i)}) + \mathbf{e}_{\text{pos}}^{(i)}$$
where $\mathbf{E} \in \mathbb{R}^{d \times P^2 C}$ is the patch embedding matrix and $\mathbf{e}_{\text{pos}}$ is the learnable 1-D positional embedding.
Classification (after $L$ encoder blocks): $$\hat{y} = \text{Linear}(\text{LN}(\mathbf{z}_L^{[CLS]}))$$
Hyperparameters for MiniViT (this chapter)¶
| Hyperparameter | Value |
|---|---|
| Image size | 32x32 |
| Patch size $P$ | 4 |
| Tokens $N+1$ | 65 |
| $d_{\text{model}}$ | 128 |
| Attention heads | 4 |
| Encoder blocks | 4 |
| FFN expansion | 2x |
| Dropout | 0.10 |
| Trainable params | ~300 K |
# -- MiniViT: patch embedding + encoder blocks + classification head ---------
class PatchEmbedding(nn.Module):
def __init__(self, img_size, patch_size, in_channels, embed_dim):
super().__init__()
self.patch_size = patch_size
n_patches = (img_size // patch_size) ** 2
flat_dim = patch_size * patch_size * in_channels
self.proj = nn.Linear(flat_dim, embed_dim)
self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
self.pos_embed = nn.Parameter(torch.zeros(1, n_patches + 1, embed_dim))
nn.init.trunc_normal_(self.cls_token, std=0.02)
nn.init.trunc_normal_(self.pos_embed, std=0.02)
def forward(self, x):
B, C, H, W = x.shape
P = self.patch_size
# unfold -> (B, C, N_H, N_W, P, P)
x = x.unfold(2, P, P).unfold(3, P, P)
# -> (B, N_H, N_W, C, P, P) -> (B, N, flat)
_, _, N_H, N_W, _, _ = x.shape
x = x.permute(0, 2, 3, 1, 4, 5).reshape(B, N_H * N_W, C * P * P)
x = self.proj(x) # (B, N, embed_dim)
cls = self.cls_token.expand(B, -1, -1)
x = torch.cat([cls, x], dim=1) # (B, N+1, embed_dim)
x = x + self.pos_embed
return x
class MHAttention(nn.Module):
def __init__(self, embed_dim, n_heads):
super().__init__()
self.n_heads = n_heads
self.head_dim = embed_dim // n_heads
self.scale = self.head_dim ** -0.5
self.qkv = nn.Linear(embed_dim, embed_dim * 3)
self.proj = nn.Linear(embed_dim, embed_dim)
def forward(self, x, return_attn=False):
B, N, D = x.shape
H = self.n_heads
qkv = self.qkv(x).reshape(B, N, 3, H, self.head_dim).permute(2, 0, 3, 1, 4)
q, k, v = qkv[0], qkv[1], qkv[2]
attn = (q @ k.transpose(-2, -1)) * self.scale
attn = attn.softmax(dim=-1)
out = (attn @ v).transpose(1, 2).reshape(B, N, D)
out = self.proj(out)
if return_attn:
return out, attn
return out
class EncoderBlock(nn.Module):
def __init__(self, embed_dim, n_heads, mlp_ratio=2, drop=0.10):
super().__init__()
self.norm1 = nn.LayerNorm(embed_dim)
self.attn = MHAttention(embed_dim, n_heads)
self.norm2 = nn.LayerNorm(embed_dim)
mlp_dim = embed_dim * mlp_ratio
self.mlp = nn.Sequential(
nn.Linear(embed_dim, mlp_dim), nn.GELU(),
nn.Dropout(drop),
nn.Linear(mlp_dim, embed_dim), nn.Dropout(drop),
)
def forward(self, x, return_attn=False):
if return_attn:
attn_out, attn_w = self.attn(self.norm1(x), return_attn=True)
x = x + attn_out
x = x + self.mlp(self.norm2(x))
return x, attn_w
x = x + self.attn(self.norm1(x))
x = x + self.mlp(self.norm2(x))
return x
class MiniViT(nn.Module):
def __init__(self, img_size=32, patch_size=4, in_channels=3,
embed_dim=128, n_heads=4, n_blocks=4, n_classes=4, mlp_ratio=2):
super().__init__()
self.patch_embed = PatchEmbedding(img_size, patch_size, in_channels, embed_dim)
self.blocks = nn.ModuleList([
EncoderBlock(embed_dim, n_heads, mlp_ratio) for _ in range(n_blocks)
])
self.norm = nn.LayerNorm(embed_dim)
self.head = nn.Linear(embed_dim, n_classes)
def forward(self, x, return_attn=False):
x = self.patch_embed(x)
attn_weights = []
for blk in self.blocks:
if return_attn:
x, aw = blk(x, return_attn=True)
attn_weights.append(aw)
else:
x = blk(x)
x = self.norm(x)
logits = self.head(x[:, 0])
if return_attn:
return logits, attn_weights
return logits
vit = MiniViT().to(device)
n_params = sum(p.numel() for p in vit.parameters() if p.requires_grad)
print(f"MiniViT trainable params : {n_params:,}")
# Verify forward pass
dummy = torch.randn(2, 3, IMG_SIZE, IMG_SIZE).to(device)
logits = vit(dummy)
print(f"Input shape : {tuple(dummy.shape)}")
print(f"Output shape : {tuple(logits.shape)}")
# Verify attention return path
logits2, attn_list = vit(dummy, return_attn=True)
print(f"Attention list length : {len(attn_list)} blocks")
print(f"Last block attn shape : {tuple(attn_list[-1].shape)}")
# Expected: (B, n_heads, N+1, N+1) = (2, 4, 65, 65)
print("[OK] MiniViT architecture built.")
§18.3 Training MiniViT on Synthetic Quadrant Images¶
To isolate the ViT mechanism, we use a synthetic task designed to require spatial attention:
- 4 classes (one per image quadrant):
- Class 0 -- Red patch in the top-left quadrant
- Class 1 -- Green patch in the top-right quadrant
- Class 2 -- Blue patch in the bottom-left quadrant
- Class 3 -- Yellow patch in the bottom-right quadrant
- Each image is 32x32 pixels. The coloured quadrant occupies exactly 16x16 pixels (4 patches x 4 patches). The remaining three quadrants are black + Gaussian noise.
- Why this task? The class-discriminating signal is perfectly spatially localised
to one quadrant. A well-trained ViT should concentrate its
[CLS]attention on the relevant quadrant -- which we verify in §18.4.
Training configuration¶
| Setting | Value |
|---|---|
| Train samples | 800 (200 per class) |
| Test samples | 200 (50 per class) |
| Optimiser | AdamW, lr=1e-3, weight_decay=0.05 |
| LR schedule | CosineAnnealingLR, T_max=10 |
| Epochs | 10 |
| Batch size | 64 |
# -- Synthetic QuadrantDataset + training loop --------------------------------
import random
from torch.utils.data import Dataset, DataLoader
random.seed(42)
torch.manual_seed(42)
rng = np.random.default_rng(42)
NOISE_STD = 0.15
def make_quadrant_image(cls_idx):
h2, w2 = IMG_SIZE // 2, IMG_SIZE // 2
img = torch.zeros(3, IMG_SIZE, IMG_SIZE)
if cls_idx == 0: # red top-left
img[0, :h2, :w2] = 1.0
elif cls_idx == 1: # green top-right
img[1, :h2, w2:] = 1.0
elif cls_idx == 2: # blue bottom-left
img[2, h2:, :w2] = 1.0
else: # yellow (red+green) bottom-right
img[0, h2:, w2:] = 1.0
img[1, h2:, w2:] = 1.0
return (img + torch.randn_like(img) * NOISE_STD).clamp(0.0, 1.0)
class QuadrantDataset(Dataset):
def __init__(self, n_per_class):
self.items = []
for cls in range(4):
for _ in range(n_per_class):
self.items.append((make_quadrant_image(cls), cls))
random.shuffle(self.items)
def __len__(self):
return len(self.items)
def __getitem__(self, i):
return self.items[i]
tr_ds = QuadrantDataset(200) # 800 train
te_ds = QuadrantDataset(50) # 200 test
tr_ld = DataLoader(tr_ds, batch_size=64, shuffle=True)
te_ld = DataLoader(te_ds, batch_size=128)
# Fresh MiniViT for training
vit2 = MiniViT().to(device)
opt = torch.optim.AdamW(vit2.parameters(), lr=1e-3, weight_decay=0.05)
sch = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=10)
EPOCHS = 10
tr_losses, te_accs = [], []
for epoch in range(1, EPOCHS + 1):
vit2.train()
ep_loss = 0.0
for imgs, labs in tr_ld:
imgs, labs = imgs.to(device), labs.to(device)
loss = F.cross_entropy(vit2(imgs), labs)
opt.zero_grad()
loss.backward()
opt.step()
ep_loss += loss.item() * imgs.size(0)
sch.step()
avg_loss = ep_loss / len(tr_ds)
vit2.eval()
preds, gts = [], []
with torch.no_grad():
for imgs, labs in te_ld:
preds.extend(vit2(imgs.to(device)).argmax(1).cpu().tolist())
gts.extend(labs.tolist())
acc = sum(p == g for p, g in zip(preds, gts)) / len(gts)
tr_losses.append(avg_loss)
te_accs.append(acc)
print(f" Epoch {epoch:2d}/{EPOCHS} loss={avg_loss:.4f} test_acc={acc:.4f}")
# Training curves
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3.5))
ax1.plot(range(1, EPOCHS + 1), tr_losses, "o-", color="steelblue", lw=2)
ax1.set_title("Training Loss"); ax1.set_xlabel("Epoch"); ax1.set_ylabel("Cross-entropy")
ax2.plot(range(1, EPOCHS + 1), [a * 100 for a in te_accs], "o-", color="darkorange", lw=2)
ax2.axhline(25, color="grey", ls="--", label="Chance (25%)")
ax2.set_title("Test Accuracy (%)"); ax2.set_xlabel("Epoch"); ax2.legend()
plt.tight_layout()
plt.show()
CLASS_NAMES = ["Red-TL", "Green-TR", "Blue-BL", "Yellow-BR"]
print(classification_report(gts, preds, target_names=CLASS_NAMES))
§18.4 Attention Map Visualisation¶
After training, we can inspect which patches the model attends to when
classifying an image. The [CLS] token aggregates information from all patch
tokens; its attention weights reveal the model's spatial focus.
Extracting [CLS] attention¶
From the last encoder block, the attention tensor has shape
$(B, H, N+1, N+1)$ where $H$ is the number of heads and $N+1$ = 65 (1 [CLS]
- 64 patches).
- Slice row 0 (the
[CLS]token) attending to columns 1: (patch tokens): $\mathbf{a} \in \mathbb{R}^{H \times N}$. - Average over heads: $\bar{\mathbf{a}} \in \mathbb{R}^{N}$.
- Reshape to the patch grid $(N_H, N_W) = (8, 8)$.
- Upsample to image resolution $(32, 32)$ via nearest-neighbour repetition.
- Overlay as a heatmap on the original image.
A correctly trained ViT should show high attention concentrated in the coloured quadrant and near-zero attention elsewhere.
# -- [CLS] attention map visualisation (last encoder block, head-averaged) ---
vit2.eval()
fig, axes = plt.subplots(2, 4, figsize=(13, 6.5))
for cls_idx in range(4):
# Find one image of this class from the test set
test_img = None
for img_t, lab_t in te_ds.items:
if lab_t == cls_idx:
test_img = img_t
break
inp = test_img.unsqueeze(0).to(device)
with torch.no_grad():
_, attn_list = vit2(inp, return_attn=True)
# Last block: (1, n_heads, N+1, N+1)
attn_last = attn_list[-1][0] # (n_heads, N+1, N+1)
cls_attn = attn_last[:, 0, 1:] # (n_heads, N_patches)
cls_attn = cls_attn.mean(0) # (N_patches,) averaged over heads
cls_attn = cls_attn.reshape(N_PH, N_PW).cpu().numpy() # (8, 8)
# Upsample to image size with nearest-neighbour (kron)
attn_map = np.kron(cls_attn, np.ones((PATCH_SIZE, PATCH_SIZE))) # (32, 32)
lo, hi = attn_map.min(), attn_map.max()
attn_map = (attn_map - lo) / (hi - lo + 1e-8)
ax_img = axes[0, cls_idx]
ax_attn = axes[1, cls_idx]
ax_img.imshow(test_img.permute(1, 2, 0).clamp(0, 1).numpy())
ax_img.set_title(CLASS_NAMES[cls_idx], fontsize=11)
ax_img.axis("off")
ax_attn.imshow(test_img.permute(1, 2, 0).clamp(0, 1).numpy())
ax_attn.imshow(attn_map, cmap="hot", alpha=0.60)
ax_attn.set_title("[CLS] attention", fontsize=11)
ax_attn.axis("off")
axes[0, 0].set_ylabel("Image", fontsize=11)
axes[1, 0].set_ylabel("[CLS] Attn", fontsize=11)
plt.suptitle(
"MiniViT [CLS] Attention Maps -- Last Encoder Block (averaged over 4 heads)\n"
"Bright = high attention; the model focuses on the discriminative quadrant.",
fontsize=11,
)
plt.tight_layout()
plt.show()
print("[OK] attention map visualisation complete.")
§18.5 Multimodal Models: CLIP and Contrastive Learning¶
What is CLIP?¶
CLIP (Contrastive Language-Image Pre-Training; Radford et al., 2021) trains two encoders jointly:
- An image encoder (ResNet or ViT) that maps images to a shared embedding space.
- A text encoder (Transformer) that maps natural-language captions to the same space.
The training objective is InfoNCE contrastive loss: given a batch of $N$ (image, text) pairs, maximise the cosine similarity of matching pairs and minimise it for the $N^2 - N$ mismatched pairs:
$$\mathcal{L}_{\text{CLIP}} = -\frac{1}{2N} \sum_{i=1}^{N} \left[ \log \frac{e^{\text{sim}(v_i, t_i) / \tau}}{\sum_j e^{\text{sim}(v_i, t_j) / \tau}} + \log \frac{e^{\text{sim}(t_i, v_i) / \tau}}{\sum_j e^{\text{sim}(t_j, v_i) / \tau}} \right]$$
where $\text{sim}(\cdot, \cdot)$ is cosine similarity and $\tau$ is a learnable temperature.
Key property: zero-shot classification¶
After training on 400 M image-text pairs (LAION-5B for open-CLIP), the model can classify any image into any set of classes by:
- Encoding class names as text:
"a photo of a {class}". - Encoding the query image.
- Returning the class whose text embedding has the highest cosine similarity.
No fine-tuning is required.
Contrastive similarity matrix¶
The ideal training signal is a $N \times N$ matrix where the diagonal equals 1 (matched pair) and off-diagonal elements approach 0 (mismatched pairs).
# -- Synthetic CLIP-style cosine similarity matrix (no model download) -------
rng = np.random.default_rng(7)
CLIP_CLASSES = [
"No nodule",
"Lung opacity",
"Cardiomegaly",
"Pleural effusion",
"Pneumothorax",
]
N_CLS = len(CLIP_CLASSES)
DIM = 64 # embedding dimension
# Simulate matched (image, text) embeddings:
# same-class pairs share a common base direction; noise breaks identity.
img_embs, txt_embs = [], []
for _ in range(N_CLS):
center = rng.standard_normal(DIM)
center /= np.linalg.norm(center)
img_embs.append(center + rng.standard_normal(DIM) * 0.05)
txt_embs.append(center + rng.standard_normal(DIM) * 0.05)
img_embs = np.stack(img_embs)
txt_embs = np.stack(txt_embs)
# L2 normalise both
img_embs /= np.linalg.norm(img_embs, axis=1, keepdims=True)
txt_embs /= np.linalg.norm(txt_embs, axis=1, keepdims=True)
sim_mat = img_embs @ txt_embs.T # (N_CLS, N_CLS)
fig, ax = plt.subplots(figsize=(6.5, 5.5))
sns.heatmap(
sim_mat, annot=True, fmt=".2f", cmap="coolwarm", center=0,
xticklabels=CLIP_CLASSES, yticklabels=CLIP_CLASSES,
linewidths=0.5, vmin=-1.0, vmax=1.0, ax=ax,
)
ax.set_title(
"Synthetic CLIP-style Image-Text Cosine Similarity\n"
"(diagonal = matched pairs; off-diagonal = mismatched)",
fontsize=10,
)
ax.set_xlabel("Text embeddings")
ax.set_ylabel("Image embeddings")
plt.xticks(rotation=30, ha="right")
plt.tight_layout()
plt.show()
diag_mean = np.diag(sim_mat).mean()
n_offdiag = N_CLS * (N_CLS - 1)
offdiag_mean = (sim_mat.sum() - np.trace(sim_mat)) / n_offdiag
print(f"Diagonal mean (matched pairs) : {diag_mean:.4f}")
print(f"Off-diagonal mean (mismatched) : {offdiag_mean:.4f}")
print(f"Contrast ratio (diag / offdiag) : {diag_mean / max(abs(offdiag_mean), 1e-6):.1f}x")
print("[OK] CLIP similarity demo complete.")
§18.6 Medical Imaging: BioViL, CheXpert, and MIMIC-CXR¶
Why ViT fits medical imaging¶
Medical images (X-rays, MRI, histopathology slides) share a key property with natural images: spatially localised pathology. A ViT's global self-attention from the first layer is particularly well-suited to detecting:
- Long-range correlations between anatomical structures.
- Subtle texture differences across large field-of-view regions.
- Relationships between lesion location and surrounding tissue context.
Key datasets¶
| Dataset | Size | Modality | Labels |
|---|---|---|---|
| CheXpert (Irvin et al., 2019) | 224 K | Chest X-ray | 14 pathology labels (multi-label) |
| MIMIC-CXR (Johnson et al., 2019) | 227 K | Chest X-ray | Radiology reports + structured labels |
| PatchCamelyon (Veeling et al., 2018) | 327 K | Histopathology | Tumour vs normal (binary) |
Domain-adapted vision-language models¶
BioViL (Bannur et al., 2023): A CLIP-style model trained on chest X-ray / report pairs from MIMIC-CXR.
- Image encoder: ResNet-50 fine-tuned on CheXpert.
- Text encoder: ClinicalBERT (Chapter 17) fine-tuned on radiology reports.
- Contrastive objective: InfoNCE on matched X-ray / impression pairs.
- Zero-shot classification: encode pathology names as text, rank by cosine similarity.
rad-DINO (Perez-Garcia et al., 2024): A DINO self-supervised ViT trained exclusively on chest X-rays. No text labels required -- learns spatial representations through self-distillation between patch views.
Practical fine-tuning pipeline¶
Pre-trained ViT (ImageNet)
|
Continue pre-training on CheXpert (domain shift)
|
Fine-tune on task: multi-label classification / report grounding
|
Evaluate on held-out MIMIC-CXR split
The key insight from Chapter 17 transfers directly: pre-train once on large unlabelled corpora, then fine-tune on small annotated datasets. The only change is that the tokeniser operates on image patches instead of word pieces.
§18.7 Summary¶
Key Concepts¶
| Concept | Description |
|---|---|
| Patch tokenisation | Divide image into $P \times P$ non-overlapping patches; flatten and project to $d_{\text{model}}$ |
| Learnable positional embedding | 1-D sequence of vectors added to patch tokens; learned jointly with model weights |
[CLS] token |
Prepended learnable token whose final hidden state represents the whole image |
| Global attention | Every patch attends to every other patch at every layer; no locality bias |
| ViT vs CNN | CNN: translation equivariance, local receptive field, data-efficient. ViT: global context, scale-hungry, benefits from pre-training |
| CLIP | Jointly trained image + text encoders with InfoNCE contrastive loss; enables zero-shot classification |
| BioViL / rad-DINO | Domain-adapted ViTs for chest X-ray; same architecture, different pre-training corpus |
Connections to Earlier Chapters¶
| Chapter | Connection |
|---|---|
| Ch14 (Neural Networks) | Cross-entropy loss, AdamW, batch training loop |
| Ch15 (CNNs) | Transfer learning paradigm; ViT replaces convolutions with self-attention |
| Ch16 (Transformers) | Encoder blocks, multi-head attention, positional encoding -- identical in ViT |
| Ch17 (BERT) | [CLS] token, pre-train/fine-tune paradigm, HuggingFace pipeline |
What's Next?¶
Chapter 19 explores Graph Neural Networks (GNNs) -- a third structural inductive bias (after locality in CNNs and sequence order in Transformers) that models data as irregular graphs. Applications include molecular property prediction, protein structure reasoning, and knowledge graph completion.
Section 18.8 — Summary & Quick Reference
Key Formulas¶
| Concept | Formula | Python |
|---|---|---|
| Patch count | $N = (H/P)^2$ for square images (e.g., $224^2/16^2 = 196$) | n = (img_size // patch_size) ** 2 |
| Patch projection | $\mathbf{e}_i = \mathbf{x}_i^{\text{patch}}\mathbf{W}_E + \mathbf{p}_i$ | nn.Linear(patch_dim, d_model) |
| Self-attention (Ch16) | $\text{Attn} = \text{softmax}(\mathbf{Q}\mathbf{K}^T/\sqrt{d_k})\mathbf{V}$ | nn.MultiheadAttention(d_model, n_heads) |
| CLIP InfoNCE loss | $\mathcal{L} = -\tfrac{1}{N}\sum_i \log \dfrac{\exp(\mathbf{v}_i\cdot\mathbf{t}_i/\tau)}{\sum_j \exp(\mathbf{v}_i\cdot\mathbf{t}_j/\tau)}$ | F.cross_entropy(logits_vi, targets) |
| Cosine similarity (Ch10) | $\cos\theta = \mathbf{a}\cdot\mathbf{b}/(\|\mathbf{a}\|\,\|\mathbf{b}\|)$ | F.cosine_similarity(img_emb, txt_emb) |
ML Connections¶
| Concept | Where It Appears |
|---|---|
| Patch as token | Images treated as sequences; same Transformer encoder as Ch16 and Ch17 |
| [CLS] token | Global image representation; identical role to BERT's [CLS] in Ch17 |
| Pre-train / fine-tune | Same paradigm as BERT (Ch17) and CNNs (Ch15); scale then specialise |
| Contrastive learning (CLIP) | InfoNCE aligns image and text embedding spaces using cosine similarity (Ch10) |
| BioViL / rad-DINO | Domain-adapted ViTs for chest X-ray; pairs ClinicalBERT (Ch17) with patch attention |
What's Next?¶
$$\boxed{\text{Ch 18: Vision Transformers}} \;\to\; \boxed{\text{Ch 19: Graph Neural Networks}} \;\to\; \boxed{\text{Ch 20: Cognitive Agent}} \;\to\; \boxed{\text{Ch 21: Bridge to Vol II}}$$
Chapter 19 introduces the third structural inductive bias in deep learning: the graph. Where CNNs assume a grid and Transformers assume a sequence, Graph Neural Networks operate on irregular topology -- atoms and bonds in molecules, residues in proteins, concepts in medical knowledge graphs. Self-attention (Ch16) is revealed to be a fully-connected GNN with learned edge weights.
References¶
- Dosovitskiy A, Beyer L, Kolesnikov A, et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021.
- Radford A, Kim JW, Hallacy C, et al. (2021). Learning transferable visual models from natural language supervision (CLIP). ICML, 139, 8748-8763.
- Bannur S, Hyland SL, Liu Q, et al. (2023). Learning to exploit temporal structure for biomedical vision-language processing (BioViL). CVPR.
- Perez-Garcia F, Sharma H, Bond-Taylor S, et al. (2024). A foundation model for clinical-grade computational pathology and rare cancers (rad-DINO). arXiv:2401.10815.
- Irvin J, Rajpurkar P, Ko M, et al. (2019). CheXpert: a large chest radiograph dataset with uncertainty labels. AAAI, 590-597.
- Johnson AEW, Pollard TJ, Berkowitz SJ, et al. (2019). MIMIC-CXR: a large publicly available database of labeled chest radiographs. arXiv:1901.07042.
- Touvron H, Cord M, Douze M, et al. (2021). Training data-efficient image transformers and distillation through attention (DeiT). ICML.
- Caron M, Touvron H, Misra I, et al. (2021). Emerging properties in self-supervised vision Transformers (DINO). ICCV, 9650-9660.
- He K, Chen X, Xie S, et al. (2022). Masked autoencoders are scalable vision learners (MAE). CVPR.
- Veeling BS, Linmans J, Winkens J, et al. (2018). Rotation equivariant CNNs for digital pathology (PatchCamelyon). MICCAI.