Statistical Basis of AI/ML \textemdash\ Attention Mechanisms and the Transformer Architecture
Chapter 16 \textemdash\ Self-Attention, Multi-Head Attention, and the PyTorch Transformer
Why This Chapter?¶
Chapter 15 showed that convolutional filters share weights across space -- detecting the same edge or texture anywhere in an image. Spatial locality is the right inductive bias for images, but it is not the right bias for sequences: the relationship between a subject and its verb may span 20 words, or a pathology report may link a finding in sentence 1 to a recommendation in sentence 6.
Self-attention solves this by letting every token attend to every other token in a single operation -- with no assumption about locality. The Transformer (Vaswani et al. 2017) stacks self-attention with feed-forward networks and skip connections into a block that is now the backbone of language models (BERT, GPT), vision models (ViT), and multimodal systems (CLIP, GPT-4V) across the clinical AI landscape.
This chapter builds the mechanism from NumPy arithmetic, then assembles the full encoder in PyTorch and trains it on a classification task.
Chapter Contents¶
| Section | Title |
|---|---|
| $\S$16.0 | Setup: Libraries and Device |
| $\S$16.1 | Scaled Dot-Product Attention |
| $\S$16.2 | Multi-Head Attention |
| $\S$16.3 | Positional Encoding |
| $\S$16.4 | The Transformer Encoder Block in PyTorch |
| $\S$16.5 | Text Classification with a Transformer |
| $\S$16.6 | Summary & Quick Reference |
Section 16.0 -- Setup: Libraries and Device
| Library | Role in this chapter |
|---|---|
numpy |
NumPy from-scratch implementations of attention |
torch |
Tensor operations, autograd, GPU compute |
torch.nn |
TransformerEncoderLayer, Embedding, Linear |
torch.nn.functional |
scaled_dot_product_attention for verification |
matplotlib / seaborn |
Attention weight heatmaps, training curves |
sklearn |
classification_report for final evaluation |
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from sklearn.metrics import classification_report
sns.set_theme(style="whitegrid", palette="muted")
torch.manual_seed(42)
rng = np.random.default_rng(42)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"PyTorch : {torch.__version__}")
print(f"Device : {device}")
if device.type == "cuda":
print(f"GPU : {torch.cuda.get_device_name(0)}")
print(f"VRAM : {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
print("Chapter 16 -- Attention and Transformers: environment ready.")
Section 16.1 -- Scaled Dot-Product Attention
Queries, Keys, and Values¶
Every input token is projected into three vectors:
$$\mathbf{Q} = \mathbf{X}\mathbf{W}^Q, \quad \mathbf{K} = \mathbf{X}\mathbf{W}^K, \quad \mathbf{V} = \mathbf{X}\mathbf{W}^V$$
- Query $\mathbf{Q}$: "what am I looking for?"
- Key $\mathbf{K}$: "what do I contain?"
- Value $\mathbf{V}$: "what do I contribute if attended to?"
The attention output for each query is a weighted sum of all values, where the weight is determined by how well the query matches each key:
$$\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \text{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}}\right)\mathbf{V}$$
The $\sqrt{d_k}$ divisor prevents the dot products from growing so large that they saturate the softmax, which would cause vanishing gradients -- exactly the same concern that motivated careful weight initialisation in Chapter 14.
Why "attention"?¶
Unlike a convolutional filter that examines a fixed local window, the softmax weights are input-dependent: which tokens a given query attends to changes with the input. A token representing "tumour" in a clinical note will attend strongly to "margin", "invasion", and "grade" -- wherever those words appear in the note.
Causal (masked) attention¶
By setting the upper triangle of the score matrix to $-\infty$ before the softmax, future tokens are invisible -- the mechanism used by autoregressive models (GPT). The encoder-only architecture (BERT) uses bidirectional attention with no mask.
# -- Scaled dot-product attention: NumPy from scratch ---------------------
def sdp_attention(Q, K, V, mask=None):
# Q, K, V: (..., L, d_k)
d_k = Q.shape[-1]
scores = (Q @ K.swapaxes(-2, -1)) / np.sqrt(d_k) # (..., L_q, L_k)
if mask is not None:
scores = np.where(mask, -1e9, scores)
e = np.exp(scores - scores.max(axis=-1, keepdims=True))
weights = e / e.sum(axis=-1, keepdims=True) # softmax row-wise
return weights @ V, weights # (..., L_q, d_k)
rng = np.random.default_rng(0)
L, d_k = 5, 8
Q = rng.standard_normal((L, d_k))
K = rng.standard_normal((L, d_k))
V = rng.standard_normal((L, d_k))
out, weights = sdp_attention(Q, K, V)
# Visualise
fig, axes = plt.subplots(1, 2, figsize=(11, 3.8))
im0 = axes[0].imshow(weights, cmap="Blues", vmin=0, vmax=weights.max())
axes[0].set_title("Attention weights (5 x 5)\nrow i attends over all keys j", fontsize=11)
axes[0].set_xlabel("Key position j"); axes[0].set_ylabel("Query position i")
for i in range(L):
for j in range(L):
c = "white" if weights[i, j] > 0.35 else "black"
axes[0].text(j, i, f"{weights[i,j]:.2f}", ha="center", va="center",
fontsize=8, color=c)
plt.colorbar(im0, ax=axes[0], fraction=0.046, pad=0.04)
im1 = axes[1].imshow(out, cmap="RdBu_r")
axes[1].set_title("Output: weighted sum of values\nshape (5, d_k=8)", fontsize=11)
axes[1].set_xlabel("d_k dimension"); axes[1].set_ylabel("Output position")
plt.colorbar(im1, ax=axes[1], fraction=0.046, pad=0.04)
plt.suptitle("Scaled dot-product attention: each query attends to all keys")
plt.tight_layout()
plt.show()
print(f"Q, K, V shape : {Q.shape}")
print(f"Score matrix : {(Q @ K.T).shape} (before scale + softmax)")
print(f"Output shape : {out.shape}")
print(f"Row sums : {weights.sum(axis=1).round(8)} (each row is a distribution)")
# Verify against PyTorch F.scaled_dot_product_attention
Q_t = torch.tensor(Q[None, ...], dtype=torch.float32) # (1, L, d_k)
K_t = torch.tensor(K[None, ...], dtype=torch.float32)
V_t = torch.tensor(V[None, ...], dtype=torch.float32)
out_pt = F.scaled_dot_product_attention(Q_t, K_t, V_t).squeeze(0).numpy()
print(f"Max abs diff vs PyTorch: {np.abs(out - out_pt).max():.2e} [OK]")
Section 16.2 -- Multi-Head Attention
Parallel Specialised Heads¶
A single attention head produces one weighted combination of values. But different linguistic or structural relationships may be best captured by different projection subspaces -- one head might track syntactic dependencies, another coreference, another proximity.
Multi-head attention runs $h$ independent attention heads in parallel, each with its own $(\mathbf{W}_i^Q, \mathbf{W}_i^K, \mathbf{W}_i^V)$ projection matrices of dimension $d_k = d_{\text{model}} / h$, then concatenates and re-projects:
$$\text{head}_i = \text{Attention}(\mathbf{X}\mathbf{W}_i^Q,\; \mathbf{X}\mathbf{W}_i^K,\; \mathbf{X}\mathbf{W}_i^V)$$
$$\text{MultiHead}(\mathbf{X}) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)\,\mathbf{W}^O$$
Because each head uses $d_k = d_{\text{model}}/h$ dimensions, the total parameter count is the same as a single head with $d_{\text{model}}$ dimensions -- but the representation is richer.
Clinical reading¶
In a clinical note encoder, one head might learn to link a drug name to its dose wherever they appear; another might link a diagnosis to its stage; another might capture negation (e.g., "no evidence of" attending back to a finding). No convolutional filter could capture all three simultaneously.
# -- Multi-head attention: NumPy from scratch --------------------------------
rng = np.random.default_rng(1)
L_s, d_m, n_h = 6, 16, 4
d_k_h = d_m // n_h
s = 1.0 / np.sqrt(d_k_h)
X_s = rng.standard_normal((L_s, d_m))
W_q = [rng.standard_normal((d_m, d_k_h)) * s for _ in range(n_h)]
W_k = [rng.standard_normal((d_m, d_k_h)) * s for _ in range(n_h)]
W_v = [rng.standard_normal((d_m, d_k_h)) * s for _ in range(n_h)]
W_o = rng.standard_normal((d_m, d_m)) * (1.0 / np.sqrt(d_m))
head_outs, head_wts = [], []
for i in range(n_h):
Q_h = X_s @ W_q[i] # (L, d_k_h)
K_h = X_s @ W_k[i]
V_h = X_s @ W_v[i]
h_out, h_wts = sdp_attention(Q_h, K_h, V_h)
head_outs.append(h_out)
head_wts.append(h_wts)
concat = np.concatenate(head_outs, axis=-1) # (L, d_m)
mha_out = concat @ W_o # (L, d_m) -- final output projection
# Visualise each head's attention pattern
fig, axes = plt.subplots(1, n_h, figsize=(12, 3.2))
for idx, (w, ax) in enumerate(zip(head_wts, axes)):
im = ax.imshow(w, cmap="Blues", vmin=0, vmax=1)
ax.set_title(f"Head {idx + 1}", fontsize=11)
ax.set_xlabel("Key pos")
if idx == 0:
ax.set_ylabel("Query pos")
plt.suptitle(
f"Multi-head attention ({n_h} heads): each head learns an independent attention pattern")
plt.tight_layout()
plt.show()
print(f"Input : {X_s.shape} ({L_s} tokens, d_model={d_m})")
print(f"Per-head d_k : {d_k_h} ({n_h} heads x {d_k_h} = {d_m})")
print(f"Concat shape : {concat.shape} (concatenate all head outputs)")
print(f"MHA output shape : {mha_out.shape} (projected back to d_model via W_O)")
Section 16.3 -- Positional Encoding
The Permutation Problem¶
Self-attention is permutation-equivariant: shuffling the input sequence produces the same output, shuffled in the same way. Attention has no inherent notion of "token 1 comes before token 2". A bag-of-words and an ordered sentence produce identical attention scores.
To inject position information, the original Transformer adds a positional encoding $\mathbf{PE}$ to the token embeddings before attention:
$$\mathbf{X}_{\text{input}} = \text{Embed}(\text{tokens}) + \mathbf{PE}$$
Sinusoidal Encoding (Vaswani et al. 2017)¶
Each position $\text{pos}$ and each embedding dimension $i$ receive:
$$PE(\text{pos},\,2i) = \sin\!\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right), \qquad PE(\text{pos},\,2i+1) = \cos\!\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)$$
Properties:
- Unique fingerprint per position (no two positions share the same encoding).
- Similarity between nearby positions is higher than between distant ones.
- Deterministic (no learned parameters) -- generalises to sequence lengths unseen during training.
Learned Positional Embeddings¶
Modern models (BERT, GPT) typically use a learnable embedding table for positions
($\mathbf{W}_{\text{pos}} \in \mathbb{R}^{L_{\max}\times d}$). Simpler to implement;
works at least as well empirically when $L_{\max}$ is fixed during training. The
TransformerClassifier in $\S$16.4 uses this approach.
# -- Sinusoidal positional encoding ----------------------------------------
def sinusoidal_pe(max_len, d_model):
pe = np.zeros((max_len, d_model))
pos = np.arange(max_len)[:, np.newaxis] # (L, 1)
div = np.exp(np.arange(0, d_model, 2) * (-np.log(10000.0) / d_model)) # (d/2,)
pe[:, 0::2] = np.sin(pos * div)
pe[:, 1::2] = np.cos(pos * div)
return pe
pe = sinusoidal_pe(50, 64)
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
im0 = axes[0].imshow(pe.T, aspect="auto", cmap="RdBu_r")
axes[0].set_xlabel("Sequence position")
axes[0].set_ylabel("Embedding dimension")
axes[0].set_title("Sinusoidal PE heat map (d=64, L=50)", fontsize=11)
plt.colorbar(im0, ax=axes[0], fraction=0.046, pad=0.04)
for i in range(4):
axes[1].plot(pe[:, i], linewidth=1.5, label=f"dim {i}")
axes[1].set_xlabel("Sequence position")
axes[1].set_ylabel("Encoding value")
axes[1].set_title("First 4 dimensions as functions of position", fontsize=11)
axes[1].legend(fontsize=9)
plt.suptitle("Positional encoding: each position gets a unique, fixed fingerprint")
plt.tight_layout()
plt.show()
# Position similarity (nearby positions are more similar)
pe_n = pe / np.linalg.norm(pe, axis=1, keepdims=True) # row-normalise
cos = pe_n @ pe_n.T
print("Cosine similarity between position 0 and position k:")
for k in [0, 1, 3, 5, 10, 20, 49]:
print(f" pos 0 vs pos {k:2d}: {cos[0, k]:.4f}")
Section 16.4 -- The Transformer Encoder Block in PyTorch
Block Structure¶
One Transformer encoder layer applies the following operations to an input sequence $\mathbf{X} \in \mathbb{R}^{L \times d}$:
$$\mathbf{Z} = \text{LayerNorm}(\mathbf{X} + \text{MultiHead}(\mathbf{X}))$$ $$\text{Output} = \text{LayerNorm}(\mathbf{Z} + \text{FFN}(\mathbf{Z}))$$
where $\text{FFN}(\mathbf{Z}) = \text{ReLU}(\mathbf{Z}\mathbf{W}_1 + \mathbf{b}_1)\mathbf{W}_2 + \mathbf{b}_2$ with $\mathbf{W}_1 \in \mathbb{R}^{d \times 4d}$ (the feed-forward dimension is conventionally $4d$).
Two skip connections (one around self-attention, one around the FFN) prevent the vanishing gradient problem observed in deep dense networks (Chapter 14, $\S$14.2). LayerNorm (applied before each sub-layer in the Pre-LN variant used here) normalises each token's $d$-dimensional representation independently -- unlike BatchNorm which normalises across the batch.
PyTorch API¶
layer = nn.TransformerEncoderLayer(d_model, nhead, dim_feedforward=4*d,
batch_first=True, norm_first=True)
encoder = nn.TransformerEncoder(layer, num_layers=N)
output = encoder(x) # x: (B, L, d) -> (B, L, d)
For classification we aggregate the sequence by mean pooling over $L$ and pass the result through a linear head -- simple, effective, and differentiable.
# -- TransformerClassifier: embed -> positional -> encoder -> pool -> head --
class TransformerClassifier(nn.Module):
def __init__(self, vocab_size, d_model, nhead, num_layers,
n_classes, max_len=32, dropout=0.1):
super().__init__()
self.tok_emb = nn.Embedding(vocab_size, d_model, padding_idx=0)
self.pos_emb = nn.Embedding(max_len, d_model) # learned positional
enc_layer = nn.TransformerEncoderLayer(
d_model = d_model,
nhead = nhead,
dim_feedforward= 4 * d_model,
dropout = dropout,
batch_first = True,
norm_first = True, # Pre-LN: more stable gradient flow
)
self.encoder = nn.TransformerEncoder(enc_layer, num_layers=num_layers)
self.drop = nn.Dropout(dropout)
self.classifier = nn.Linear(d_model, n_classes)
def forward(self, x, pad_mask=None):
B, L = x.shape
pos = torch.arange(L, device=x.device).unsqueeze(0).expand(B, -1)
x = self.drop(self.tok_emb(x) + self.pos_emb(pos))
x = self.encoder(x, src_key_padding_mask=pad_mask)
x = x.mean(dim=1) # mean-pool over sequence length
return self.classifier(x)
# Instantiate with small hyper-parameters for the demo task
clf = TransformerClassifier(
vocab_size=200, d_model=32, nhead=4, num_layers=2,
n_classes=3, max_len=22, dropout=0.1,
).to(device)
print(clf)
print()
n_params = sum(p.numel() for p in clf.parameters())
print(f"Total parameters: {n_params:,}")
# Shape sanity check
dummy = torch.randint(1, 200, (8, 20), device=device) # batch=8, seq_len=20
out = clf(dummy)
print(f"Input : {tuple(dummy.shape)} (batch=8, seq_len=20)")
print(f"Output : {tuple(out.shape)} (batch=8, n_classes=3) [OK]")
Section 16.5 -- Text Classification with a Transformer
Synthetic Clinical-Vocabulary Dataset¶
Training on real clinical notes requires de-identification and data-use agreements. Instead we construct a synthetic dataset that captures the key statistical structure:
- Vocabulary of 200 tokens (think of each token as a clinical concept).
- Three classes, each "owning" roughly one third of the vocabulary (e.g., Class 0 ~ haematology terms, Class 1 ~ oncology terms, Class 2 ~ radiology terms).
- Each training sequence has 20 tokens: 70 % drawn from its class's vocabulary range, 30 % sampled uniformly as noise (simulating shared medical terminology).
- 1 500 training sequences, 500 test sequences.
A purely lexical classifier (count class-range tokens, take the majority) would do well here. The transformer must discover this structure through self-attention and back-propagation -- the same mechanics that allow BERT to learn from clinical notes.
Training Protocol¶
| Hyper-parameter | Value | Rationale |
|---|---|---|
| Optimiser | AdamW, $\eta = 3\times10^{-4}$, $\lambda=10^{-2}$ | Weight decay regularises the embedding table |
| Schedule | OneCycleLR | Warm-up then cosine decay; stable for small data |
| Gradient clip | 1.0 | Prevents rare large-gradient steps from destabilising attention |
| Epochs | 15 | Sufficient for this synthetic task with small model |
# -- Synthetic dataset construction ----------------------------------------
rng = np.random.default_rng(42)
torch.manual_seed(42)
VOCAB = 200
SEQ_L = 20
N_CLS = 3
SIGNAL = 0.70 # fraction of tokens from class-specific vocab
BOUNDARIES = np.linspace(1, VOCAB, N_CLS + 1, dtype=int) # [1, 67, 134, 200]
def make_split(n_total):
Xs, ys = [], []
per_cls = n_total // N_CLS
for cls in range(N_CLS):
lo, hi = int(BOUNDARIES[cls]), int(BOUNDARIES[cls + 1])
for _ in range(per_cls):
n_sig = int(SEQ_L * SIGNAL)
sig = rng.integers(lo, hi, size=n_sig)
noise = rng.integers(1, VOCAB, size=SEQ_L - n_sig)
seq = np.concatenate([sig, noise])
rng.shuffle(seq)
Xs.append(seq)
ys.append(cls)
return np.array(Xs, dtype=np.int64), np.array(ys, dtype=np.int64)
X_tr, y_tr = make_split(1500)
X_te, y_te = make_split(500)
print(f"Training : {X_tr.shape} class counts: {np.bincount(y_tr)}")
print(f"Test : {X_te.shape} class counts: {np.bincount(y_te)}")
# -- DataLoaders ------------------------------------------------------------
tr_ds = torch.utils.data.TensorDataset(
torch.from_numpy(X_tr), torch.from_numpy(y_tr))
te_ds = torch.utils.data.TensorDataset(
torch.from_numpy(X_te), torch.from_numpy(y_te))
tr_ld = torch.utils.data.DataLoader(tr_ds, batch_size=64, shuffle=True)
te_ld = torch.utils.data.DataLoader(te_ds, batch_size=128, shuffle=False)
# -- Model, optimiser, scheduler -------------------------------------------
model = TransformerClassifier(
vocab_size=VOCAB, d_model=32, nhead=4, num_layers=2,
n_classes=N_CLS, max_len=SEQ_L + 2, dropout=0.1,
).to(device)
crit = nn.CrossEntropyLoss()
opt = optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-2)
sched = optim.lr_scheduler.OneCycleLR(
opt, max_lr=3e-4, steps_per_epoch=len(tr_ld), epochs=15)
# -- Training loop ---------------------------------------------------------
EPOCHS = 15
history = {"loss": [], "acc": []}
CLASS_NAMES = ["Class A", "Class B", "Class C"]
for epoch in range(1, EPOCHS + 1):
model.train()
run_loss = 0.0
for Xb, yb in tr_ld:
Xb, yb = Xb.to(device), yb.to(device)
opt.zero_grad()
loss = crit(model(Xb), yb)
loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()
sched.step()
run_loss += loss.item() * Xb.size(0)
history["loss"].append(run_loss / len(tr_ds))
model.eval()
correct = 0
with torch.no_grad():
for Xb, yb in te_ld:
correct += (model(Xb.to(device)).argmax(1).cpu() == yb).sum().item()
history["acc"].append(correct / len(te_ds))
if epoch % 5 == 0 or epoch == 1:
print(f" Epoch {epoch:2d}/{EPOCHS} "
f"loss={history['loss'][-1]:.4f} "
f"test_acc={history['acc'][-1]:.4f}")
# -- Plots -----------------------------------------------------------------
fig, axes = plt.subplots(1, 2, figsize=(11, 3.5))
axes[0].plot(history["loss"], marker="o", markersize=4, color="steelblue")
axes[0].set_title("Training loss"); axes[0].set_xlabel("Epoch")
axes[0].set_ylabel("Cross-entropy loss")
axes[1].plot([a * 100 for a in history["acc"]], marker="s", markersize=4,
color="darkorange")
axes[1].set_title("Test accuracy"); axes[1].set_xlabel("Epoch")
axes[1].set_ylabel("Accuracy (%)")
plt.suptitle(
f"Transformer encoder: 3-class classification | device: {device}", fontsize=11)
plt.tight_layout()
plt.show()
# -- Classification report -------------------------------------------------
model.eval()
all_p, all_y = [], []
with torch.no_grad():
for Xb, yb in te_ld:
all_p.extend(model(Xb.to(device)).argmax(1).cpu().numpy())
all_y.extend(yb.numpy())
print(classification_report(all_y, all_p, target_names=CLASS_NAMES, digits=3))
Section 16.6 -- Summary & Quick Reference
Key Formulas¶
| Concept | Formula | Section |
|---|---|---|
| Scaled dot-product attention | $\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \text{softmax}\!\left(\dfrac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}}\right)\mathbf{V}$ | $\S$16.1 |
| Attention score scaling | $/ \sqrt{d_k}$ prevents softmax saturation | $\S$16.1 |
| Multi-head attention | $\text{MultiHead}(\mathbf{X}) = \text{Concat}(\text{head}_1,\ldots,\text{head}_h)\mathbf{W}^O$ | $\S$16.2 |
| Parameter equivalence | $h$ heads of $d_k=d/h$ cost same FLOPs as one head of $d$ | $\S$16.2 |
| Sinusoidal PE | $PE(\text{pos},2i)=\sin(\text{pos}/10000^{2i/d})$ | $\S$16.3 |
| Encoder block (Pre-LN) | $\mathbf{Z}=\text{LN}(\mathbf{X}+\text{MHA}(\mathbf{X}));\;\text{Out}=\text{LN}(\mathbf{Z}+\text{FFN}(\mathbf{Z}))$ | $\S$16.4 |
| Mean pooling | $\mathbf{h} = \frac{1}{L}\sum_{t=1}^{L}\mathbf{z}_t$ for sequence classification | $\S$16.5 |
ML Connections¶
| Concept | Broader context |
|---|---|
| Q, K, V projections | Three learned linear maps; the same algebra as any hidden layer (Ch 14) |
| Skip connections | Identical to ResNet's residual $\mathbf{a}^{[l+1]}=f(\mathbf{z})+\mathbf{a}^{[l]}$ (Ch 15) |
| LayerNorm | Normalises activations per token (cf. BatchNorm per feature across a batch) |
| AdamW | Adam + decoupled weight decay; standard choice for Transformer training |
| Permutation equivariance | Attention ignores order; positional encoding restores it |
| Pre-training + fine-tuning | Encoder trained on large corpus (BERT/BioBERT), fine-tuned on task data (Ch 15 transfer learning) |
What's Next?¶
$$\boxed{\text{Ch16: Attention \& Transformers} \;\longrightarrow\; \text{BERT, HuggingFace, and Clinical NLP}}$$
This chapter built the Transformer encoder from first principles. The next chapter applies pretrained encoder models to real text:
- BERT (Devlin et al. 2019) pre-trains a 12-layer bidirectional Transformer on 800 M words using masked language modelling; fine-tuning on a new task requires only a few hundred labelled examples.
- BioBERT (Lee et al. 2020) and ClinicalBERT (Alsentzer et al. 2019) adapt BERT to biomedical and clinical note text respectively.
- The HuggingFace
transformerslibrary provides one-line access to thousands of pretrained models and a standardised training API (Trainer). - Clinical NLP tasks: ICD coding, named-entity recognition (medications, diagnoses, procedures), de-identification, and clinical trial eligibility.
References¶
Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. NeurIPS 2017. The original Transformer paper; introduced scaled dot-product and multi-head attention. https://arxiv.org/abs/1706.03762
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT 2019. Bidirectional encoder pre-trained with masked language modelling. https://arxiv.org/abs/1810.04805
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. CVPR 2016. Residual connections (skip connections) used in every Transformer block. https://arxiv.org/abs/1512.03385
Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv:1607.06450. LayerNorm: normalises each token's representation independently. https://arxiv.org/abs/1607.06450
Lee, J., Yoon, W., Kim, S., et al. (2020). BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4), 1234-1240. https://doi.org/10.1093/bioinformatics/btz682
Alsentzer, E., Murphy, J., Boag, W., et al. (2019). Publicly available clinical BERT embeddings. NAACL Clinical NLP Workshop 2019. ClinicalBERT: BERT fine-tuned on MIMIC-III clinical notes. https://arxiv.org/abs/1904.03323
Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021. Vision Transformer (ViT): extends the Transformer to image patches. https://arxiv.org/abs/2010.11929
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapters 10-11 (sequence models) and background on attention. https://www.deeplearningbook.org/
Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. ICLR 2019. AdamW optimiser used throughout this chapter. https://arxiv.org/abs/1711.05101
Smith, L. N., & Topin, N. (2019). Super-convergence: Very fast training of neural networks using large learning rates. SPIE 2019. OneCycleLR schedule: warm-up + cosine decay. https://arxiv.org/abs/1708.07120