Statistical Basis of AI/ML¶
BERT, HuggingFace Transformers, and Clinical NLP¶
Chapter 17 -- Pre-training, Fine-tuning, and Domain Adaptation¶
From the Vaswani (2017) architecture to BERT's bidirectional pre-training, to fine-tuning on downstream clinical tasks -- this chapter bridges the theoretical attention mechanism from Chapter 16 to real-world NLP practice.
Why This Chapter?¶
Chapter 16 built the Transformer encoder from scratch using NumPy and PyTorch. This chapter shows how the same architecture is industrialised via the HuggingFace Transformers library, and why pre-training on large corpora followed by fine-tuning on small task-specific datasets has become the dominant paradigm in NLP -- including clinical NLP.
| Section | Topic |
|---|---|
| §17.0 | Setup and library overview |
| §17.1 | BERT architecture and pre-training objectives |
| §17.2 | Tokenization with AutoTokenizer |
| §17.3 | Feature extraction: [CLS] embeddings |
| §17.4 | Fine-tuning for sequence classification |
| §17.5 | The HuggingFace pipeline API |
| §17.6 | Clinical NLP: BioBERT and ClinicalBERT |
| §17.7 | Summary and references |
§17.0 Setup¶
| Library | Role in this chapter |
|---|---|
transformers |
AutoTokenizer, AutoModel, AutoModelForSequenceClassification, pipeline |
torch |
GPU tensors, DataLoader, AdamW optimiser, gradient clipping |
numpy |
Numerical utilities |
matplotlib / seaborn |
Training curves, embedding visualisation |
sklearn |
classification_report, PCA for embedding plots |
All model downloads are cached at %USERPROFILE%\.cache\huggingface\hub
after the first run.
# -- Imports ------------------------------------------------------------------
import warnings
warnings.filterwarnings("ignore")
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import torch
import transformers
from transformers import (
AutoTokenizer,
AutoModel,
AutoModelForSequenceClassification,
pipeline as hf_pipeline,
)
from sklearn.metrics import classification_report
from sklearn.decomposition import PCA
sns.set_theme(style="whitegrid", font_scale=1.05)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Transformers : {transformers.__version__}")
print(f"PyTorch : {torch.__version__}")
print(f"Device : {device}")
if device.type == "cuda":
print(f"GPU : {torch.cuda.get_device_name(0)}")
vram = torch.cuda.get_device_properties(0).total_memory / 1e9
print(f"VRAM : {vram:.1f} GB")
print("Chapter 17 -- BERT and HuggingFace: environment ready.")
§17.1 BERT: Bidirectional Encoder Representations from Transformers¶
BERT (Devlin et al., 2019) uses the Transformer encoder (identical to the blocks built in Chapter 16) but introduces two key ideas:
1. Bidirectional pre-training¶
Earlier LMs (ELMo, GPT) read text left-to-right or right-to-left. BERT reads both directions simultaneously using two self-supervised objectives:
Masked Language Modelling (MLM): 15% of tokens are randomly replaced with
[MASK]. The model predicts the original token from bidirectional context:$$\text{loss}_{\text{MLM}} = -\sum_{i \in \mathcal{M}} \log P(x_i \mid x_{\setminus i})$$
Next Sentence Prediction (NSP): Given two segments A and B, predict whether B follows A in the original document. Later work (Liu et al., 2019) found NSP less useful; RoBERTa drops it entirely.
2. Special tokens¶
| Token | Role |
|---|---|
[CLS] |
Prepended to every input; its final hidden state is the sentence-level representation used for classification |
[SEP] |
Separates two segments (sentence A from sentence B) |
[MASK] |
Placeholder for masked tokens during MLM pre-training |
[PAD] |
Padding to reach uniform sequence length |
Architecture comparison¶
| Component | BERT-base | DistilBERT |
|---|---|---|
| Encoder layers | 12 | 6 |
| Hidden size $d_{\text{model}}$ | 768 | 768 |
| Attention heads | 12 | 12 |
| Parameters | ~110 M | ~66 M |
| Pre-training data | 16 GB | distilled from BERT |
We use DistilBERT in this chapter: 40% smaller, 60% faster, and retaining 97% of BERT's performance on GLUE benchmarks (Sanh et al., 2019).
# -- Tokenization: AutoTokenizer (DistilBERT) ---------------------------------
MODEL_NAME = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
sentences = [
"The patient shows no evidence of malignant transformation.",
"High-grade invasive carcinoma with perineural invasion.",
]
for sent in sentences:
enc = tokenizer(sent, return_tensors="pt")
tokens = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])
print(f"Text : {sent}")
print(f"Tokens : {tokens}")
print(f"IDs : {enc['input_ids'][0].tolist()}")
print()
# Batch encoding with padding and truncation
batch = tokenizer(
sentences, padding=True, truncation=True, max_length=32, return_tensors="pt"
)
print("Batch input_ids shape :", batch["input_ids"].shape)
print("Batch attention_mask shape:", batch["attention_mask"].shape)
print(f"Vocabulary size : {tokenizer.vocab_size:,}")
print("[OK] tokenizer ready.")
§17.2 Tokenization and Sub-word Representations¶
WordPiece tokenisation (used by BERT and DistilBERT) splits rare words into
frequent sub-word units, marked with ##:
"perineural" -> ['peri', '##neur', '##al']
"carcinoma" -> ['car', '##cin', '##oma']
This gives three advantages over word-level tokenisation:
- Fixed vocabulary (30,522 tokens for BERT-base) -- any input is representable without unknown-token collisions.
- Morphological sharing: "invasive", "invasion", "invasively" all share
the stem
invasiin the vocabulary. - Biomedical robustness: clinical abbreviations and drug names that miss the full-word vocabulary are always decomposable into known sub-words.
The [CLS] embedding¶
After a forward pass through the model, last_hidden_state[:, 0, :] extracts
the hidden state at position 0 (the [CLS] token), yielding a
$d_{\text{model}}$-dimensional vector.
- Pre-trained (not fine-tuned): generic sentence representation.
- After fine-tuning on a task: the
[CLS]vector becomes task-specific, and its geometry reflects the decision boundaries of the downstream labels.
# -- Feature extraction: [CLS] embeddings from the pre-trained backbone ------
backbone = AutoModel.from_pretrained(MODEL_NAME).to(device)
backbone.eval()
fe_sentences = [
"No evidence of malignant nodule on imaging.", # benign
"Margins are clear with benign histology.", # benign
"Invasive carcinoma with lymph node involvement.", # malignant
"Metastatic lesion confirmed by biopsy.", # malignant
]
with torch.no_grad():
enc_fe = tokenizer(
fe_sentences, padding=True, truncation=True,
max_length=64, return_tensors="pt"
).to(device)
out_fe = backbone(**enc_fe)
cls_embs = out_fe.last_hidden_state[:, 0, :].cpu() # (4, 768)
print(f"[CLS] embedding shape : {tuple(cls_embs.shape)}")
# Cosine similarity between sentence pairs
from torch.nn.functional import cosine_similarity as cos_sim
pairs = [
(0, 1, "benign vs benign "),
(0, 2, "benign vs malignant"),
(2, 3, "malignant vs malignant"),
]
for i, j, label in pairs:
sim = cos_sim(cls_embs[i:i+1], cls_embs[j:j+1]).item()
print(f" cos({label}) = {sim:.4f}")
print("[OK] pre-trained [CLS] embeddings extracted.")
§17.3 Fine-tuning for Sequence Classification¶
The pre-train / fine-tune paradigm has three steps:
- Pre-train once on massive unlabelled corpora (Wikipedia, BooksCorpus, PubMed). This is expensive and done once by a research lab.
- Download the pre-trained checkpoint:
AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=k). - Fine-tune on your labelled downstream task -- often a small dataset (hundreds to thousands of examples), a few epochs, and a small learning rate (~2e-5) to avoid catastrophic forgetting of pre-trained weights.
Input: [CLS] word1 word2 ... [SEP]
|
DistilBERT (6 encoder layers)
|
[CLS] hidden state (768-dim)
|
Linear(768 -> n_classes)
|
Softmax -> class probabilities
Practical fine-tuning guidelines¶
| Hyperparameter | Typical range | Rationale |
|---|---|---|
| Learning rate | 1e-5 -- 5e-5 | Above ~1e-4 risks catastrophic forgetting |
| Batch size | 16 -- 32 | GPU memory and gradient stability |
| Epochs | 2 -- 5 | Quick convergence; risk of overfitting on small data |
| Weight decay | 0.01 | Standard AdamW regularisation |
| Gradient clipping | 1.0 | Prevents rare large gradient spikes |
# -- Fine-tuning DistilBERT on synthetic clinical text (3-class) -------------
import random
from torch.utils.data import Dataset, DataLoader
random.seed(42)
torch.manual_seed(42)
# Synthetic dataset -- Benign / Malignant / Uncertain
TEMPLATES = {
0: [ # Benign / normal
"No evidence of {t} on imaging.",
"Margins are clear with {t} histology.",
"Follow-up shows stable {t} without progression.",
"The {t} appears benign with well-defined borders.",
"No abnormal {t} detected on current workup.",
"Resolved {t} with complete clinical response.",
],
1: [ # Malignant
"Invasive {t} with lymph node involvement.",
"Metastatic {t} confirmed by biopsy.",
"Aggressive {t} with perineural invasion.",
"Malignant {t} requiring immediate intervention.",
"High-grade {t} with poor differentiation.",
"Recurrent {t} with extensive local infiltration.",
],
2: [ # Uncertain
"Indeterminate {t} requiring further evaluation.",
"The {t} is inconclusive; repeat imaging recommended.",
"Uncertain {t} -- clinical correlation advised.",
"Pending {t} results; differential remains broad.",
"Equivocal {t} on current workup.",
"The {t} is atypical; additional sampling required.",
],
}
TERMS = ["lesion", "mass", "nodule", "opacity", "enhancement", "density", "signal"]
def make_dataset(n_per_class):
texts, labels = [], []
for cls, tmpls in TEMPLATES.items():
for _ in range(n_per_class):
texts.append(random.choice(tmpls).format(t=random.choice(TERMS)))
labels.append(cls)
pairs = list(zip(texts, labels))
random.shuffle(pairs)
return [p[0] for p in pairs], [p[1] for p in pairs]
train_texts, train_labels = make_dataset(200) # 600 train
test_texts, test_labels = make_dataset(50) # 150 test
MAX_LEN = 64
tr_enc = tokenizer(
train_texts, padding="max_length", truncation=True,
max_length=MAX_LEN, return_tensors="pt"
)
te_enc = tokenizer(
test_texts, padding="max_length", truncation=True,
max_length=MAX_LEN, return_tensors="pt"
)
class ClinDataset(Dataset):
def __init__(self, enc, labs):
self.enc = enc
self.labs = torch.tensor(labs, dtype=torch.long)
def __len__(self):
return len(self.labs)
def __getitem__(self, i):
return {k: v[i] for k, v in self.enc.items()}, self.labs[i]
tr_ld = DataLoader(ClinDataset(tr_enc, train_labels), batch_size=32, shuffle=True)
te_ld = DataLoader(ClinDataset(te_enc, test_labels), batch_size=64)
# Load DistilBERT with 3-class classification head
clf = AutoModelForSequenceClassification.from_pretrained(
MODEL_NAME, num_labels=3
).to(device)
n_p = sum(p.numel() for p in clf.parameters() if p.requires_grad)
print(f"Trainable params: {n_p:,}")
opt = torch.optim.AdamW(clf.parameters(), lr=2e-5, weight_decay=0.01)
EPOCHS = 4
tr_losses, te_accs = [], []
for epoch in range(1, EPOCHS + 1):
clf.train()
ep_loss = 0.0
for bx, by in tr_ld:
bx = {k: v.to(device) for k, v in bx.items()}
by = by.to(device)
result = clf(**bx, labels=by)
opt.zero_grad()
result.loss.backward()
torch.nn.utils.clip_grad_norm_(clf.parameters(), 1.0)
opt.step()
ep_loss += result.loss.item() * by.size(0)
avg_loss = ep_loss / len(train_labels)
clf.eval()
preds, gts = [], []
with torch.no_grad():
for bx, by in te_ld:
bx = {k: v.to(device) for k, v in bx.items()}
preds.extend(clf(**bx).logits.argmax(dim=1).cpu().tolist())
gts.extend(by.tolist())
acc = sum(p == g for p, g in zip(preds, gts)) / len(gts)
tr_losses.append(avg_loss)
te_accs.append(acc)
print(f" Epoch {epoch}/{EPOCHS} loss={avg_loss:.4f} test_acc={acc:.4f}")
# Training curves
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3.5))
ax1.plot(range(1, EPOCHS + 1), tr_losses, "o-", color="steelblue", lw=2)
ax1.set_title("Training Loss"); ax1.set_xlabel("Epoch"); ax1.set_ylabel("Cross-entropy")
ax2.plot(range(1, EPOCHS + 1), [a * 100 for a in te_accs], "o-", color="darkorange", lw=2)
ax2.axhline(100 / 3, color="grey", ls="--", label="Chance (33%)")
ax2.set_title("Test Accuracy (%)"); ax2.set_xlabel("Epoch")
ax2.legend(); plt.tight_layout(); plt.show()
CLASS_NAMES = ["Benign", "Malignant", "Uncertain"]
print(classification_report(gts, preds, target_names=CLASS_NAMES))
§17.4 The HuggingFace pipeline API¶
pipeline wraps tokenisation + model forward pass + post-processing into a
single callable, enabling one-line inference:
from transformers import pipeline
pipe = pipeline("text-classification",
model="distilbert-base-uncased-finetuned-sst-2-english")
pipe("I love this!") # -> [{'label': 'POSITIVE', 'score': 0.9998}]
The pipeline function wraps any PreTrainedModel, including a model you
just fine-tuned in the current session. Requirements:
- Set
model.config.id2labelto map integer indices to readable label names. - Pass
model=clf, tokenizer=tokenizertopipeline().
Available task identifiers¶
| Task string | Application |
|---|---|
"text-classification" |
Sentiment, topic, clinical triage |
"token-classification" |
NER, part-of-speech tagging |
"question-answering" |
Extractive QA (SQuAD-style) |
"zero-shot-classification" |
No-label-needed classification |
"fill-mask" |
Predict masked tokens (BERT-style) |
"summarization" |
Abstractive summarisation |
# -- HuggingFace pipeline API (wrapping our fine-tuned classifier) -----------
CLASS_NAMES = ["Benign", "Malignant", "Uncertain"]
clf.config.id2label = {i: n for i, n in enumerate(CLASS_NAMES)}
clf.config.label2id = {n: i for i, n in enumerate(CLASS_NAMES)}
clf_pipe = hf_pipeline(
task="text-classification",
model=clf,
tokenizer=tokenizer,
device=0 if device.type == "cuda" else -1,
)
test_phrases = [
"No evidence of malignant nodule on imaging.",
"Invasive carcinoma with lymph node involvement.",
"Indeterminate enhancement requiring further evaluation.",
"Resolved opacity with complete clinical response.",
"Equivocal density -- clinical correlation advised.",
]
print(f"{'Prediction':>12} {'Score':>6} Text")
print("-" * 72)
for phrase in test_phrases:
res = clf_pipe(phrase, truncation=True, max_length=MAX_LEN)[0]
print(f" {res['label']:>10} {res['score']:.4f} {phrase}")
print()
print("[OK] pipeline inference complete.")
§17.5 Clinical NLP: BioBERT and ClinicalBERT¶
General-domain BERT was pre-trained on Wikipedia and BooksCorpus. Clinical text has very different statistical properties:
| Property | General text | Clinical text |
|---|---|---|
| Vocabulary | Everyday English | Medical abbreviations, drug names, ICD codes |
| Sentence structure | Grammatically complete | Telegraphic ("No e/o mets.") |
| Named entities | People, places, organisations | Diseases, drugs, procedures, genes |
| Negation | Rare ambiguity | Critical ("no evidence of", "denies") |
| Sub-word fragmentation | Low | High (clinical terms split heavily) |
BioBERT (Lee et al., 2020)¶
Fine-tuned BERT on PubMed abstracts (4.5 B words) and PMC full-text articles (13.5 B words). State-of-the-art on biomedical NER, relation extraction, and QA tasks.
HuggingFace Hub: dmis-lab/biobert-base-cased-v1.1
ClinicalBERT (Alsentzer et al., 2019)¶
Continued pre-training of BioBERT on ~2 M MIMIC-III clinical notes (ICU records from Beth Israel Deaconess Medical Center). Captures clinical abbreviations, lab values, and nursing narrative.
HuggingFace Hub: emilyalsentzer/Bio_ClinicalBERT
Domain adaptation strategy¶
General BERT ---(continue pre-train on PubMed)---> BioBERT
BioBERT ---(continue pre-train on MIMIC)----> ClinicalBERT
ClinicalBERT ---(fine-tune on task labels)-------> Task model
Each arrow applies the same Transformer encoder architecture and masked language modelling objective; only the pre-training corpus changes.
# -- PCA visualisation of fine-tuned [CLS] embeddings (test set) -------------
clf.eval()
all_embs, all_labs = [], []
with torch.no_grad():
for bx, by in te_ld:
bx = {k: v.to(device) for k, v in bx.items()}
# Access the DistilBERT backbone directly to get last_hidden_state
out_v = clf.distilbert(**bx)
cls_v = out_v.last_hidden_state[:, 0, :].cpu().numpy()
all_embs.append(cls_v)
all_labs.extend(by.tolist())
all_embs = np.concatenate(all_embs, axis=0) # (150, 768)
all_labs = np.array(all_labs)
pca = PCA(n_components=2, random_state=42)
emb2d = pca.fit_transform(all_embs)
var1, var2 = pca.explained_variance_ratio_
print(f"PCA explained variance: PC1={var1:.3f} PC2={var2:.3f} "
f"total={var1 + var2:.3f}")
fig, ax = plt.subplots(figsize=(7, 5.5))
colors = ["steelblue", "firebrick", "darkorange"]
for cls_idx, (cname, col) in enumerate(zip(CLASS_NAMES, colors)):
mask = all_labs == cls_idx
ax.scatter(emb2d[mask, 0], emb2d[mask, 1], c=col, label=cname,
alpha=0.75, edgecolors="white", linewidths=0.4, s=60)
ax.set_title("PCA of Fine-tuned [CLS] Embeddings (test set)", fontsize=13)
ax.set_xlabel(f"PC1 ({var1:.1%} var.)")
ax.set_ylabel(f"PC2 ({var2:.1%} var.)")
ax.legend(title="Class", framealpha=0.9)
plt.tight_layout(); plt.show()
print("[OK] PCA embedding visualisation complete.")
§17.6 Summary¶
Key Concepts¶
| Concept | Description |
|---|---|
| WordPiece tokenisation | Sub-word vocabulary (30,522 tokens); any word decomposable |
| MLM pre-training | Predict 15% masked tokens from full bidirectional context |
[CLS] embedding |
Sentence-level representation; task-specific after fine-tuning |
| Fine-tuning | Update all weights with small LR on labelled downstream data |
pipeline API |
One-call inference: tokenise + forward pass + label decode |
| Domain adaptation | Continue MLM pre-training on target-domain text corpus |
| BioBERT / ClinicalBERT | Domain-adapted checkpoints for PubMed / ICU clinical notes |
Connections to Earlier Chapters¶
| Chapter | Connection |
|---|---|
| Ch07 (Logistic Regression) | Linear classification head on top of [CLS] embedding |
| Ch06 / Ch14 (Regularisation) | Weight decay in AdamW prevents fine-tuning overfitting |
| Ch14 (Neural Networks) | Multi-layer stacked encoder blocks, cross-entropy loss |
| Ch15 (CNNs) | Transfer learning / fine-tuning paradigm is identical |
| Ch16 (Transformers) | BERT IS the Transformer encoder built in Ch16 |
What's Next?¶
Chapter 18 will explore Vision Transformers (ViT) -- applying the identical attention mechanism to image patches -- and multimodal models that combine image and text representations for tasks such as medical imaging report generation.
Section 17.7 — Summary & Quick Reference
Key Formulas¶
| Concept | Formula | Python |
|---|---|---|
| MLM pre-training loss | $\mathcal{L}_{\text{MLM}} = -\!\sum_{m \in \mathcal{M}} \log P(x_m \mid x_{\backslash m};\,\theta)$ | AutoModelForMaskedLM |
| Fine-tuning CE loss | $\mathcal{L} = -\!\sum_i y_i \log \hat{p}_i$ | nn.CrossEntropyLoss() |
| [CLS] classification head | $\hat{\mathbf{p}} = \text{softmax}(\mathbf{W}\mathbf{h}_{[\text{CLS}]} + \mathbf{b})$ | AutoModelForSequenceClassification |
| AdamW update | $\theta \leftarrow \theta(1-\eta\lambda) - \eta\,\hat{\mathbf{m}}/(\sqrt{\hat{\mathbf{v}}}+\varepsilon)$ | AdamW(lr=2e-5, weight_decay=0.01) |
| WordPiece vocab | 30,522 tokens (BERT-base); sub-word decomposition | AutoTokenizer.from_pretrained(...) |
ML Connections¶
| Concept | Where It Appears |
|---|---|
| Pre-training / fine-tuning | Same paradigm as CNNs in Ch15; large-corpus initialisation, then task-specific adaptation |
| [CLS] embedding | Sentence-level feature vector; input to any downstream classifier (Ch07, Ch14) |
| Weight decay (AdamW) | L2 regularisation on weights (Ch06, Ch14); prevents overfitting on small fine-tuning sets |
| Cross-entropy loss | Same loss used in every classifier from Ch07 onward and in Ch14 neural networks |
| Domain adaptation | BioBERT, ClinicalBERT -- continue MLM pre-training on target corpus; same math, new data |
What's Next?¶
$$\boxed{\text{Ch 17: BERT}} \;\to\; \boxed{\text{Ch 18: Vision Transformers}} \;\to\; \boxed{\text{Ch 19: Graph Neural Networks}} \;\to\; \boxed{\text{Ch 20: Cognitive Agent}}$$
Chapter 18 applies the identical Transformer encoder to images: each image is divided into fixed-size patches, each patch is projected to a $d_{\text{model}}$-dimensional embedding, and the resulting sequence is processed by the same multi-head attention blocks built in Chapter 16. The only difference is what gets tokenised.
References¶
- Devlin J, Chang M-W, Lee K, Toutanova K (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT, 4171-4186.
- Sanh V, Debut L, Chaumond J, Wolf T (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108.
- Lee J, Yoon W, Kim S, et al. (2020). BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4), 1234-1240.
- Alsentzer E, Murphy J, Boag W, et al. (2019). Publicly available clinical BERT embeddings. NAACL Clinical NLP Workshop, 72-78.
- Vaswani A, Shazeer N, Parmar N, et al. (2017). Attention is all you need. NeurIPS, 30.
- Liu Y, Ott M, Goyal N, et al. (2019). RoBERTa: a robustly optimized BERT pretraining approach. arXiv:1907.11692.
- Wolf T, Debut L, Sanh V, et al. (2020). HuggingFace's Transformers: state-of-the-art natural language processing. EMNLP Systems, 38-45.
- Loshchilov I, Hutter F (2019). Decoupled weight decay regularisation. ICLR.
- Johnson AEW, Pollard TJ, Shen L, et al. (2016). MIMIC-III, a freely accessible critical care database. Scientific Data, 3, 160035.
- Goodfellow I, Bengio Y, Courville A (2016). Deep Learning. MIT Press.