Home / About Us / Volume I Contents / Ch 16: Attention and Transformers

Statistical Basis of AI/ML \textemdash\ Attention Mechanisms and the Transformer Architecture

Chapter 16 \textemdash\ Self-Attention, Multi-Head Attention, and the PyTorch Transformer


Why This Chapter?

Chapter 15 showed that convolutional filters share weights across space -- detecting the same edge or texture anywhere in an image. Spatial locality is the right inductive bias for images, but it is not the right bias for sequences: the relationship between a subject and its verb may span 20 words, or a pathology report may link a finding in sentence 1 to a recommendation in sentence 6.

Self-attention solves this by letting every token attend to every other token in a single operation -- with no assumption about locality. The Transformer (Vaswani et al. 2017) stacks self-attention with feed-forward networks and skip connections into a block that is now the backbone of language models (BERT, GPT), vision models (ViT), and multimodal systems (CLIP, GPT-4V) across the clinical AI landscape.

This chapter builds the mechanism from NumPy arithmetic, then assembles the full encoder in PyTorch and trains it on a classification task.

Chapter Contents

Section Title
$\S$16.0 Setup: Libraries and Device
$\S$16.1 Scaled Dot-Product Attention
$\S$16.2 Multi-Head Attention
$\S$16.3 Positional Encoding
$\S$16.4 The Transformer Encoder Block in PyTorch
$\S$16.5 Text Classification with a Transformer
$\S$16.6 Summary & Quick Reference

Section 16.0 -- Setup: Libraries and Device

Library Role in this chapter
numpy NumPy from-scratch implementations of attention
torch Tensor operations, autograd, GPU compute
torch.nn TransformerEncoderLayer, Embedding, Linear
torch.nn.functional scaled_dot_product_attention for verification
matplotlib / seaborn Attention weight heatmaps, training curves
sklearn classification_report for final evaluation
In [1]:
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from sklearn.metrics import classification_report

sns.set_theme(style="whitegrid", palette="muted")
torch.manual_seed(42)
rng = np.random.default_rng(42)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

print(f"PyTorch  : {torch.__version__}")
print(f"Device   : {device}")
if device.type == "cuda":
    print(f"GPU      : {torch.cuda.get_device_name(0)}")
    print(f"VRAM     : {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
print("Chapter 16 -- Attention and Transformers: environment ready.")
PyTorch  : 2.11.0+cu128
Device   : cuda
GPU      : NVIDIA GeForce RTX 5080
VRAM     : 17.1 GB
Chapter 16 -- Attention and Transformers: environment ready.

Section 16.1 -- Scaled Dot-Product Attention

Queries, Keys, and Values

Every input token is projected into three vectors:

$$\mathbf{Q} = \mathbf{X}\mathbf{W}^Q, \quad \mathbf{K} = \mathbf{X}\mathbf{W}^K, \quad \mathbf{V} = \mathbf{X}\mathbf{W}^V$$

  • Query $\mathbf{Q}$: "what am I looking for?"
  • Key $\mathbf{K}$: "what do I contain?"
  • Value $\mathbf{V}$: "what do I contribute if attended to?"

The attention output for each query is a weighted sum of all values, where the weight is determined by how well the query matches each key:

$$\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \text{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}}\right)\mathbf{V}$$

The $\sqrt{d_k}$ divisor prevents the dot products from growing so large that they saturate the softmax, which would cause vanishing gradients -- exactly the same concern that motivated careful weight initialisation in Chapter 14.

Why "attention"?

Unlike a convolutional filter that examines a fixed local window, the softmax weights are input-dependent: which tokens a given query attends to changes with the input. A token representing "tumour" in a clinical note will attend strongly to "margin", "invasion", and "grade" -- wherever those words appear in the note.

Causal (masked) attention

By setting the upper triangle of the score matrix to $-\infty$ before the softmax, future tokens are invisible -- the mechanism used by autoregressive models (GPT). The encoder-only architecture (BERT) uses bidirectional attention with no mask.

In [2]:
# -- Scaled dot-product attention: NumPy from scratch ---------------------
def sdp_attention(Q, K, V, mask=None):
    # Q, K, V: (..., L, d_k)
    d_k = Q.shape[-1]
    scores = (Q @ K.swapaxes(-2, -1)) / np.sqrt(d_k)    # (..., L_q, L_k)
    if mask is not None:
        scores = np.where(mask, -1e9, scores)
    e = np.exp(scores - scores.max(axis=-1, keepdims=True))
    weights = e / e.sum(axis=-1, keepdims=True)            # softmax row-wise
    return weights @ V, weights                            # (..., L_q, d_k)


rng = np.random.default_rng(0)
L, d_k = 5, 8

Q = rng.standard_normal((L, d_k))
K = rng.standard_normal((L, d_k))
V = rng.standard_normal((L, d_k))

out, weights = sdp_attention(Q, K, V)

# Visualise
fig, axes = plt.subplots(1, 2, figsize=(11, 3.8))

im0 = axes[0].imshow(weights, cmap="Blues", vmin=0, vmax=weights.max())
axes[0].set_title("Attention weights (5 x 5)\nrow i attends over all keys j", fontsize=11)
axes[0].set_xlabel("Key position j"); axes[0].set_ylabel("Query position i")
for i in range(L):
    for j in range(L):
        c = "white" if weights[i, j] > 0.35 else "black"
        axes[0].text(j, i, f"{weights[i,j]:.2f}", ha="center", va="center",
                     fontsize=8, color=c)
plt.colorbar(im0, ax=axes[0], fraction=0.046, pad=0.04)

im1 = axes[1].imshow(out, cmap="RdBu_r")
axes[1].set_title("Output: weighted sum of values\nshape (5, d_k=8)", fontsize=11)
axes[1].set_xlabel("d_k dimension"); axes[1].set_ylabel("Output position")
plt.colorbar(im1, ax=axes[1], fraction=0.046, pad=0.04)

plt.suptitle("Scaled dot-product attention: each query attends to all keys")
plt.tight_layout()
plt.show()

print(f"Q, K, V shape  : {Q.shape}")
print(f"Score matrix   : {(Q @ K.T).shape}  (before scale + softmax)")
print(f"Output shape   : {out.shape}")
print(f"Row sums       : {weights.sum(axis=1).round(8)}  (each row is a distribution)")

# Verify against PyTorch F.scaled_dot_product_attention
Q_t = torch.tensor(Q[None, ...], dtype=torch.float32)   # (1, L, d_k)
K_t = torch.tensor(K[None, ...], dtype=torch.float32)
V_t = torch.tensor(V[None, ...], dtype=torch.float32)
out_pt = F.scaled_dot_product_attention(Q_t, K_t, V_t).squeeze(0).numpy()
print(f"Max abs diff vs PyTorch: {np.abs(out - out_pt).max():.2e}  [OK]")
No description has been provided for this image
Q, K, V shape  : (5, 8)
Score matrix   : (5, 5)  (before scale + softmax)
Output shape   : (5, 8)
Row sums       : [1. 1. 1. 1. 1.]  (each row is a distribution)
Max abs diff vs PyTorch: 1.77e-07  [OK]

Section 16.2 -- Multi-Head Attention

Parallel Specialised Heads

A single attention head produces one weighted combination of values. But different linguistic or structural relationships may be best captured by different projection subspaces -- one head might track syntactic dependencies, another coreference, another proximity.

Multi-head attention runs $h$ independent attention heads in parallel, each with its own $(\mathbf{W}_i^Q, \mathbf{W}_i^K, \mathbf{W}_i^V)$ projection matrices of dimension $d_k = d_{\text{model}} / h$, then concatenates and re-projects:

$$\text{head}_i = \text{Attention}(\mathbf{X}\mathbf{W}_i^Q,\; \mathbf{X}\mathbf{W}_i^K,\; \mathbf{X}\mathbf{W}_i^V)$$

$$\text{MultiHead}(\mathbf{X}) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)\,\mathbf{W}^O$$

Because each head uses $d_k = d_{\text{model}}/h$ dimensions, the total parameter count is the same as a single head with $d_{\text{model}}$ dimensions -- but the representation is richer.

Clinical reading

In a clinical note encoder, one head might learn to link a drug name to its dose wherever they appear; another might link a diagnosis to its stage; another might capture negation (e.g., "no evidence of" attending back to a finding). No convolutional filter could capture all three simultaneously.

In [3]:
# -- Multi-head attention: NumPy from scratch --------------------------------
rng = np.random.default_rng(1)
L_s, d_m, n_h = 6, 16, 4
d_k_h = d_m // n_h
s = 1.0 / np.sqrt(d_k_h)

X_s = rng.standard_normal((L_s, d_m))

W_q = [rng.standard_normal((d_m, d_k_h)) * s for _ in range(n_h)]
W_k = [rng.standard_normal((d_m, d_k_h)) * s for _ in range(n_h)]
W_v = [rng.standard_normal((d_m, d_k_h)) * s for _ in range(n_h)]
W_o = rng.standard_normal((d_m, d_m)) * (1.0 / np.sqrt(d_m))

head_outs, head_wts = [], []
for i in range(n_h):
    Q_h = X_s @ W_q[i]    # (L, d_k_h)
    K_h = X_s @ W_k[i]
    V_h = X_s @ W_v[i]
    h_out, h_wts = sdp_attention(Q_h, K_h, V_h)
    head_outs.append(h_out)
    head_wts.append(h_wts)

concat  = np.concatenate(head_outs, axis=-1)  # (L, d_m)
mha_out = concat @ W_o                         # (L, d_m) -- final output projection

# Visualise each head's attention pattern
fig, axes = plt.subplots(1, n_h, figsize=(12, 3.2))
for idx, (w, ax) in enumerate(zip(head_wts, axes)):
    im = ax.imshow(w, cmap="Blues", vmin=0, vmax=1)
    ax.set_title(f"Head {idx + 1}", fontsize=11)
    ax.set_xlabel("Key pos")
    if idx == 0:
        ax.set_ylabel("Query pos")
plt.suptitle(
    f"Multi-head attention ({n_h} heads): each head learns an independent attention pattern")
plt.tight_layout()
plt.show()

print(f"Input            : {X_s.shape}  ({L_s} tokens, d_model={d_m})")
print(f"Per-head d_k     : {d_k_h}  ({n_h} heads x {d_k_h} = {d_m})")
print(f"Concat shape     : {concat.shape}  (concatenate all head outputs)")
print(f"MHA output shape : {mha_out.shape}  (projected back to d_model via W_O)")
No description has been provided for this image
Input            : (6, 16)  (6 tokens, d_model=16)
Per-head d_k     : 4  (4 heads x 4 = 16)
Concat shape     : (6, 16)  (concatenate all head outputs)
MHA output shape : (6, 16)  (projected back to d_model via W_O)

Section 16.3 -- Positional Encoding

The Permutation Problem

Self-attention is permutation-equivariant: shuffling the input sequence produces the same output, shuffled in the same way. Attention has no inherent notion of "token 1 comes before token 2". A bag-of-words and an ordered sentence produce identical attention scores.

To inject position information, the original Transformer adds a positional encoding $\mathbf{PE}$ to the token embeddings before attention:

$$\mathbf{X}_{\text{input}} = \text{Embed}(\text{tokens}) + \mathbf{PE}$$

Sinusoidal Encoding (Vaswani et al. 2017)

Each position $\text{pos}$ and each embedding dimension $i$ receive:

$$PE(\text{pos},\,2i) = \sin\!\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right), \qquad PE(\text{pos},\,2i+1) = \cos\!\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$$

Properties:

  • Unique fingerprint per position (no two positions share the same encoding).
  • Similarity between nearby positions is higher than between distant ones.
  • Deterministic (no learned parameters) -- generalises to sequence lengths unseen during training.

Learned Positional Embeddings

Modern models (BERT, GPT) typically use a learnable embedding table for positions ($\mathbf{W}_{\text{pos}} \in \mathbb{R}^{L_{\max}\times d}$). Simpler to implement; works at least as well empirically when $L_{\max}$ is fixed during training. The TransformerClassifier in $\S$16.4 uses this approach.

In [4]:
# -- Sinusoidal positional encoding ----------------------------------------
def sinusoidal_pe(max_len, d_model):
    pe  = np.zeros((max_len, d_model))
    pos = np.arange(max_len)[:, np.newaxis]                          # (L, 1)
    div = np.exp(np.arange(0, d_model, 2) * (-np.log(10000.0) / d_model))  # (d/2,)
    pe[:, 0::2] = np.sin(pos * div)
    pe[:, 1::2] = np.cos(pos * div)
    return pe


pe = sinusoidal_pe(50, 64)

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

im0 = axes[0].imshow(pe.T, aspect="auto", cmap="RdBu_r")
axes[0].set_xlabel("Sequence position")
axes[0].set_ylabel("Embedding dimension")
axes[0].set_title("Sinusoidal PE heat map (d=64, L=50)", fontsize=11)
plt.colorbar(im0, ax=axes[0], fraction=0.046, pad=0.04)

for i in range(4):
    axes[1].plot(pe[:, i], linewidth=1.5, label=f"dim {i}")
axes[1].set_xlabel("Sequence position")
axes[1].set_ylabel("Encoding value")
axes[1].set_title("First 4 dimensions as functions of position", fontsize=11)
axes[1].legend(fontsize=9)

plt.suptitle("Positional encoding: each position gets a unique, fixed fingerprint")
plt.tight_layout()
plt.show()

# Position similarity (nearby positions are more similar)
pe_n = pe / np.linalg.norm(pe, axis=1, keepdims=True)   # row-normalise
cos  = pe_n @ pe_n.T

print("Cosine similarity between position 0 and position k:")
for k in [0, 1, 3, 5, 10, 20, 49]:
    print(f"  pos 0 vs pos {k:2d}: {cos[0, k]:.4f}")
No description has been provided for this image
Cosine similarity between position 0 and position k:
  pos 0 vs pos  0: 1.0000
  pos 0 vs pos  1: 0.9662
  pos 0 vs pos  3: 0.7996
  pos 0 vs pos  5: 0.7345
  pos 0 vs pos 10: 0.6579
  pos 0 vs pos 20: 0.5900
  pos 0 vs pos 49: 0.4754

Section 16.4 -- The Transformer Encoder Block in PyTorch

Block Structure

One Transformer encoder layer applies the following operations to an input sequence $\mathbf{X} \in \mathbb{R}^{L \times d}$:

$$\mathbf{Z} = \text{LayerNorm}(\mathbf{X} + \text{MultiHead}(\mathbf{X}))$$ $$\text{Output} = \text{LayerNorm}(\mathbf{Z} + \text{FFN}(\mathbf{Z}))$$

where $\text{FFN}(\mathbf{Z}) = \text{ReLU}(\mathbf{Z}\mathbf{W}_1 + \mathbf{b}_1)\mathbf{W}_2 + \mathbf{b}_2$ with $\mathbf{W}_1 \in \mathbb{R}^{d \times 4d}$ (the feed-forward dimension is conventionally $4d$).

Two skip connections (one around self-attention, one around the FFN) prevent the vanishing gradient problem observed in deep dense networks (Chapter 14, $\S$14.2). LayerNorm (applied before each sub-layer in the Pre-LN variant used here) normalises each token's $d$-dimensional representation independently -- unlike BatchNorm which normalises across the batch.

PyTorch API

layer  = nn.TransformerEncoderLayer(d_model, nhead, dim_feedforward=4*d,
                                    batch_first=True, norm_first=True)
encoder = nn.TransformerEncoder(layer, num_layers=N)
output  = encoder(x)          # x: (B, L, d)  ->  (B, L, d)

For classification we aggregate the sequence by mean pooling over $L$ and pass the result through a linear head -- simple, effective, and differentiable.

In [5]:
# -- TransformerClassifier: embed -> positional -> encoder -> pool -> head --
class TransformerClassifier(nn.Module):
    def __init__(self, vocab_size, d_model, nhead, num_layers,
                 n_classes, max_len=32, dropout=0.1):
        super().__init__()
        self.tok_emb = nn.Embedding(vocab_size, d_model, padding_idx=0)
        self.pos_emb = nn.Embedding(max_len, d_model)   # learned positional
        enc_layer = nn.TransformerEncoderLayer(
            d_model        = d_model,
            nhead          = nhead,
            dim_feedforward= 4 * d_model,
            dropout        = dropout,
            batch_first    = True,
            norm_first     = True,   # Pre-LN: more stable gradient flow
        )
        self.encoder    = nn.TransformerEncoder(enc_layer, num_layers=num_layers)
        self.drop       = nn.Dropout(dropout)
        self.classifier = nn.Linear(d_model, n_classes)

    def forward(self, x, pad_mask=None):
        B, L = x.shape
        pos  = torch.arange(L, device=x.device).unsqueeze(0).expand(B, -1)
        x    = self.drop(self.tok_emb(x) + self.pos_emb(pos))
        x    = self.encoder(x, src_key_padding_mask=pad_mask)
        x    = x.mean(dim=1)         # mean-pool over sequence length
        return self.classifier(x)


# Instantiate with small hyper-parameters for the demo task
clf = TransformerClassifier(
    vocab_size=200, d_model=32, nhead=4, num_layers=2,
    n_classes=3, max_len=22, dropout=0.1,
).to(device)

print(clf)
print()
n_params = sum(p.numel() for p in clf.parameters())
print(f"Total parameters: {n_params:,}")

# Shape sanity check
dummy = torch.randint(1, 200, (8, 20), device=device)   # batch=8, seq_len=20
out   = clf(dummy)
print(f"Input  : {tuple(dummy.shape)}  (batch=8, seq_len=20)")
print(f"Output : {tuple(out.shape)}   (batch=8, n_classes=3)  [OK]")
TransformerClassifier(
  (tok_emb): Embedding(200, 32, padding_idx=0)
  (pos_emb): Embedding(22, 32)
  (encoder): TransformerEncoder(
    (layers): ModuleList(
      (0-1): 2 x TransformerEncoderLayer(
        (self_attn): MultiheadAttention(
          (out_proj): NonDynamicallyQuantizableLinear(in_features=32, out_features=32, bias=True)
        )
        (linear1): Linear(in_features=32, out_features=128, bias=True)
        (dropout): Dropout(p=0.1, inplace=False)
        (linear2): Linear(in_features=128, out_features=32, bias=True)
        (norm1): LayerNorm((32,), eps=1e-05, elementwise_affine=True)
        (norm2): LayerNorm((32,), eps=1e-05, elementwise_affine=True)
        (dropout1): Dropout(p=0.1, inplace=False)
        (dropout2): Dropout(p=0.1, inplace=False)
      )
    )
  )
  (drop): Dropout(p=0.1, inplace=False)
  (classifier): Linear(in_features=32, out_features=3, bias=True)
)

Total parameters: 32,611
/tmp/ipykernel_719/3976623047.py:16: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True
  self.encoder    = nn.TransformerEncoder(enc_layer, num_layers=num_layers)
Input  : (8, 20)  (batch=8, seq_len=20)
Output : (8, 3)   (batch=8, n_classes=3)  [OK]

Section 16.5 -- Text Classification with a Transformer

Synthetic Clinical-Vocabulary Dataset

Training on real clinical notes requires de-identification and data-use agreements. Instead we construct a synthetic dataset that captures the key statistical structure:

  • Vocabulary of 200 tokens (think of each token as a clinical concept).
  • Three classes, each "owning" roughly one third of the vocabulary (e.g., Class 0 ~ haematology terms, Class 1 ~ oncology terms, Class 2 ~ radiology terms).
  • Each training sequence has 20 tokens: 70 % drawn from its class's vocabulary range, 30 % sampled uniformly as noise (simulating shared medical terminology).
  • 1 500 training sequences, 500 test sequences.

A purely lexical classifier (count class-range tokens, take the majority) would do well here. The transformer must discover this structure through self-attention and back-propagation -- the same mechanics that allow BERT to learn from clinical notes.

Training Protocol

Hyper-parameter Value Rationale
Optimiser AdamW, $\eta = 3\times10^{-4}$, $\lambda=10^{-2}$ Weight decay regularises the embedding table
Schedule OneCycleLR Warm-up then cosine decay; stable for small data
Gradient clip 1.0 Prevents rare large-gradient steps from destabilising attention
Epochs 15 Sufficient for this synthetic task with small model
In [6]:
# -- Synthetic dataset construction ----------------------------------------
rng = np.random.default_rng(42)
torch.manual_seed(42)

VOCAB  = 200
SEQ_L  = 20
N_CLS  = 3
SIGNAL = 0.70   # fraction of tokens from class-specific vocab

BOUNDARIES = np.linspace(1, VOCAB, N_CLS + 1, dtype=int)   # [1, 67, 134, 200]

def make_split(n_total):
    Xs, ys = [], []
    per_cls = n_total // N_CLS
    for cls in range(N_CLS):
        lo, hi = int(BOUNDARIES[cls]), int(BOUNDARIES[cls + 1])
        for _ in range(per_cls):
            n_sig   = int(SEQ_L * SIGNAL)
            sig     = rng.integers(lo, hi, size=n_sig)
            noise   = rng.integers(1, VOCAB, size=SEQ_L - n_sig)
            seq     = np.concatenate([sig, noise])
            rng.shuffle(seq)
            Xs.append(seq)
            ys.append(cls)
    return np.array(Xs, dtype=np.int64), np.array(ys, dtype=np.int64)

X_tr, y_tr = make_split(1500)
X_te, y_te = make_split(500)

print(f"Training : {X_tr.shape}  class counts: {np.bincount(y_tr)}")
print(f"Test     : {X_te.shape}  class counts: {np.bincount(y_te)}")

# -- DataLoaders ------------------------------------------------------------
tr_ds = torch.utils.data.TensorDataset(
    torch.from_numpy(X_tr), torch.from_numpy(y_tr))
te_ds = torch.utils.data.TensorDataset(
    torch.from_numpy(X_te), torch.from_numpy(y_te))
tr_ld = torch.utils.data.DataLoader(tr_ds, batch_size=64, shuffle=True)
te_ld = torch.utils.data.DataLoader(te_ds, batch_size=128, shuffle=False)

# -- Model, optimiser, scheduler -------------------------------------------
model = TransformerClassifier(
    vocab_size=VOCAB, d_model=32, nhead=4, num_layers=2,
    n_classes=N_CLS, max_len=SEQ_L + 2, dropout=0.1,
).to(device)

crit  = nn.CrossEntropyLoss()
opt   = optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-2)
sched = optim.lr_scheduler.OneCycleLR(
    opt, max_lr=3e-4, steps_per_epoch=len(tr_ld), epochs=15)

# -- Training loop ---------------------------------------------------------
EPOCHS  = 15
history = {"loss": [], "acc": []}
CLASS_NAMES = ["Class A", "Class B", "Class C"]

for epoch in range(1, EPOCHS + 1):
    model.train()
    run_loss = 0.0
    for Xb, yb in tr_ld:
        Xb, yb = Xb.to(device), yb.to(device)
        opt.zero_grad()
        loss = crit(model(Xb), yb)
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
        sched.step()
        run_loss += loss.item() * Xb.size(0)
    history["loss"].append(run_loss / len(tr_ds))

    model.eval()
    correct = 0
    with torch.no_grad():
        for Xb, yb in te_ld:
            correct += (model(Xb.to(device)).argmax(1).cpu() == yb).sum().item()
    history["acc"].append(correct / len(te_ds))

    if epoch % 5 == 0 or epoch == 1:
        print(f"  Epoch {epoch:2d}/{EPOCHS}  "
              f"loss={history['loss'][-1]:.4f}  "
              f"test_acc={history['acc'][-1]:.4f}")

# -- Plots -----------------------------------------------------------------
fig, axes = plt.subplots(1, 2, figsize=(11, 3.5))
axes[0].plot(history["loss"], marker="o", markersize=4, color="steelblue")
axes[0].set_title("Training loss"); axes[0].set_xlabel("Epoch")
axes[0].set_ylabel("Cross-entropy loss")

axes[1].plot([a * 100 for a in history["acc"]], marker="s", markersize=4,
             color="darkorange")
axes[1].set_title("Test accuracy"); axes[1].set_xlabel("Epoch")
axes[1].set_ylabel("Accuracy (%)")

plt.suptitle(
    f"Transformer encoder: 3-class classification  |  device: {device}", fontsize=11)
plt.tight_layout()
plt.show()

# -- Classification report -------------------------------------------------
model.eval()
all_p, all_y = [], []
with torch.no_grad():
    for Xb, yb in te_ld:
        all_p.extend(model(Xb.to(device)).argmax(1).cpu().numpy())
        all_y.extend(yb.numpy())
print(classification_report(all_y, all_p, target_names=CLASS_NAMES, digits=3))
Training : (1500, 20)  class counts: [500 500 500]
Test     : (498, 20)  class counts: [166 166 166]
/tmp/ipykernel_719/3976623047.py:16: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True
  self.encoder    = nn.TransformerEncoder(enc_layer, num_layers=num_layers)
  Epoch  1/15  loss=1.1189  test_acc=0.3133
  Epoch  5/15  loss=0.8955  test_acc=0.8353
  Epoch 10/15  loss=0.1803  test_acc=0.9438
  Epoch 15/15  loss=0.1276  test_acc=0.9558
No description has been provided for this image
              precision    recall  f1-score   support

     Class A      0.964     0.964     0.964       166
     Class B      0.975     0.940     0.957       166
     Class C      0.930     0.964     0.947       166

    accuracy                          0.956       498
   macro avg      0.956     0.956     0.956       498
weighted avg      0.956     0.956     0.956       498


Section 16.6 -- Summary & Quick Reference

Key Formulas

Concept Formula Section
Scaled dot-product attention $\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \text{softmax}\!\left(\dfrac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}}\right)\mathbf{V}$ $\S$16.1
Attention score scaling $/ \sqrt{d_k}$ prevents softmax saturation $\S$16.1
Multi-head attention $\text{MultiHead}(\mathbf{X}) = \text{Concat}(\text{head}_1,\ldots,\text{head}_h)\mathbf{W}^O$ $\S$16.2
Parameter equivalence $h$ heads of $d_k=d/h$ cost same FLOPs as one head of $d$ $\S$16.2
Sinusoidal PE $PE(\text{pos},2i)=\sin(\text{pos}/10000^{2i/d})$ $\S$16.3
Encoder block (Pre-LN) $\mathbf{Z}=\text{LN}(\mathbf{X}+\text{MHA}(\mathbf{X}));\;\text{Out}=\text{LN}(\mathbf{Z}+\text{FFN}(\mathbf{Z}))$ $\S$16.4
Mean pooling $\mathbf{h} = \frac{1}{L}\sum_{t=1}^{L}\mathbf{z}_t$ for sequence classification $\S$16.5

ML Connections

Concept Broader context
Q, K, V projections Three learned linear maps; the same algebra as any hidden layer (Ch 14)
Skip connections Identical to ResNet's residual $\mathbf{a}^{[l+1]}=f(\mathbf{z})+\mathbf{a}^{[l]}$ (Ch 15)
LayerNorm Normalises activations per token (cf. BatchNorm per feature across a batch)
AdamW Adam + decoupled weight decay; standard choice for Transformer training
Permutation equivariance Attention ignores order; positional encoding restores it
Pre-training + fine-tuning Encoder trained on large corpus (BERT/BioBERT), fine-tuned on task data (Ch 15 transfer learning)

What's Next?

$$\boxed{\text{Ch16: Attention \& Transformers} \;\longrightarrow\; \text{BERT, HuggingFace, and Clinical NLP}}$$

This chapter built the Transformer encoder from first principles. The next chapter applies pretrained encoder models to real text:

  • BERT (Devlin et al. 2019) pre-trains a 12-layer bidirectional Transformer on 800 M words using masked language modelling; fine-tuning on a new task requires only a few hundred labelled examples.
  • BioBERT (Lee et al. 2020) and ClinicalBERT (Alsentzer et al. 2019) adapt BERT to biomedical and clinical note text respectively.
  • The HuggingFace transformers library provides one-line access to thousands of pretrained models and a standardised training API (Trainer).
  • Clinical NLP tasks: ICD coding, named-entity recognition (medications, diagnoses, procedures), de-identification, and clinical trial eligibility.

References

  1. Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. NeurIPS 2017. The original Transformer paper; introduced scaled dot-product and multi-head attention. https://arxiv.org/abs/1706.03762

  2. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT 2019. Bidirectional encoder pre-trained with masked language modelling. https://arxiv.org/abs/1810.04805

  3. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. CVPR 2016. Residual connections (skip connections) used in every Transformer block. https://arxiv.org/abs/1512.03385

  4. Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv:1607.06450. LayerNorm: normalises each token's representation independently. https://arxiv.org/abs/1607.06450

  5. Lee, J., Yoon, W., Kim, S., et al. (2020). BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4), 1234-1240. https://doi.org/10.1093/bioinformatics/btz682

  6. Alsentzer, E., Murphy, J., Boag, W., et al. (2019). Publicly available clinical BERT embeddings. NAACL Clinical NLP Workshop 2019. ClinicalBERT: BERT fine-tuned on MIMIC-III clinical notes. https://arxiv.org/abs/1904.03323

  7. Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021. Vision Transformer (ViT): extends the Transformer to image patches. https://arxiv.org/abs/2010.11929

  8. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapters 10-11 (sequence models) and background on attention. https://www.deeplearningbook.org/

  9. Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. ICLR 2019. AdamW optimiser used throughout this chapter. https://arxiv.org/abs/1711.05101

  10. Smith, L. N., & Topin, N. (2019). Super-convergence: Very fast training of neural networks using large learning rates. SPIE 2019. OneCycleLR schedule: warm-up + cosine decay. https://arxiv.org/abs/1708.07120