Home / About Us / Volume I Contents / Ch 17: BERT and HuggingFace

Statistical Basis of AI/ML

BERT, HuggingFace Transformers, and Clinical NLP

Chapter 17 -- Pre-training, Fine-tuning, and Domain Adaptation

From the Vaswani (2017) architecture to BERT's bidirectional pre-training, to fine-tuning on downstream clinical tasks -- this chapter bridges the theoretical attention mechanism from Chapter 16 to real-world NLP practice.

Why This Chapter?

Chapter 16 built the Transformer encoder from scratch using NumPy and PyTorch. This chapter shows how the same architecture is industrialised via the HuggingFace Transformers library, and why pre-training on large corpora followed by fine-tuning on small task-specific datasets has become the dominant paradigm in NLP -- including clinical NLP.

Section Topic
§17.0 Setup and library overview
§17.1 BERT architecture and pre-training objectives
§17.2 Tokenization with AutoTokenizer
§17.3 Feature extraction: [CLS] embeddings
§17.4 Fine-tuning for sequence classification
§17.5 The HuggingFace pipeline API
§17.6 Clinical NLP: BioBERT and ClinicalBERT
§17.7 Summary and references

§17.0 Setup

Library Role in this chapter
transformers AutoTokenizer, AutoModel, AutoModelForSequenceClassification, pipeline
torch GPU tensors, DataLoader, AdamW optimiser, gradient clipping
numpy Numerical utilities
matplotlib / seaborn Training curves, embedding visualisation
sklearn classification_report, PCA for embedding plots

All model downloads are cached at %USERPROFILE%\.cache\huggingface\hub after the first run.

In [1]:
# -- Imports ------------------------------------------------------------------
import warnings
warnings.filterwarnings("ignore")

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import torch
import transformers
from transformers import (
    AutoTokenizer,
    AutoModel,
    AutoModelForSequenceClassification,
    pipeline as hf_pipeline,
)
from sklearn.metrics import classification_report
from sklearn.decomposition import PCA

sns.set_theme(style="whitegrid", font_scale=1.05)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

print(f"Transformers : {transformers.__version__}")
print(f"PyTorch      : {torch.__version__}")
print(f"Device       : {device}")
if device.type == "cuda":
    print(f"GPU          : {torch.cuda.get_device_name(0)}")
    vram = torch.cuda.get_device_properties(0).total_memory / 1e9
    print(f"VRAM         : {vram:.1f} GB")
print("Chapter 17 -- BERT and HuggingFace: environment ready.")
Transformers : 5.8.0
PyTorch      : 2.11.0+cu128
Device       : cuda
GPU          : NVIDIA GeForce RTX 5080
VRAM         : 17.1 GB
Chapter 17 -- BERT and HuggingFace: environment ready.

§17.1 BERT: Bidirectional Encoder Representations from Transformers

BERT (Devlin et al., 2019) uses the Transformer encoder (identical to the blocks built in Chapter 16) but introduces two key ideas:

1. Bidirectional pre-training

Earlier LMs (ELMo, GPT) read text left-to-right or right-to-left. BERT reads both directions simultaneously using two self-supervised objectives:

  • Masked Language Modelling (MLM): 15% of tokens are randomly replaced with [MASK]. The model predicts the original token from bidirectional context:

    $$\text{loss}_{\text{MLM}} = -\sum_{i \in \mathcal{M}} \log P(x_i \mid x_{\setminus i})$$

  • Next Sentence Prediction (NSP): Given two segments A and B, predict whether B follows A in the original document. Later work (Liu et al., 2019) found NSP less useful; RoBERTa drops it entirely.

2. Special tokens

Token Role
[CLS] Prepended to every input; its final hidden state is the sentence-level representation used for classification
[SEP] Separates two segments (sentence A from sentence B)
[MASK] Placeholder for masked tokens during MLM pre-training
[PAD] Padding to reach uniform sequence length

Architecture comparison

Component BERT-base DistilBERT
Encoder layers 12 6
Hidden size $d_{\text{model}}$ 768 768
Attention heads 12 12
Parameters ~110 M ~66 M
Pre-training data 16 GB distilled from BERT

We use DistilBERT in this chapter: 40% smaller, 60% faster, and retaining 97% of BERT's performance on GLUE benchmarks (Sanh et al., 2019).

In [2]:
# -- Tokenization: AutoTokenizer (DistilBERT) ---------------------------------
MODEL_NAME = "distilbert-base-uncased"
tokenizer  = AutoTokenizer.from_pretrained(MODEL_NAME)

sentences = [
    "The patient shows no evidence of malignant transformation.",
    "High-grade invasive carcinoma with perineural invasion.",
]

for sent in sentences:
    enc    = tokenizer(sent, return_tensors="pt")
    tokens = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])
    print(f"Text   : {sent}")
    print(f"Tokens : {tokens}")
    print(f"IDs    : {enc['input_ids'][0].tolist()}")
    print()

# Batch encoding with padding and truncation
batch = tokenizer(
    sentences, padding=True, truncation=True, max_length=32, return_tensors="pt"
)
print("Batch input_ids shape     :", batch["input_ids"].shape)
print("Batch attention_mask shape:", batch["attention_mask"].shape)
print(f"Vocabulary size           : {tokenizer.vocab_size:,}")
print("[OK] tokenizer ready.")
Text   : The patient shows no evidence of malignant transformation.
Tokens : ['[CLS]', 'the', 'patient', 'shows', 'no', 'evidence', 'of', 'mali', '##gnant', 'transformation', '.', '[SEP]']
IDs    : [101, 1996, 5776, 3065, 2053, 3350, 1997, 16007, 27881, 8651, 1012, 102]

Text   : High-grade invasive carcinoma with perineural invasion.
Tokens : ['[CLS]', 'high', '-', 'grade', 'invasive', 'car', '##cino', '##ma', 'with', 'per', '##ine', '##ural', 'invasion', '.', '[SEP]']
IDs    : [101, 2152, 1011, 3694, 17503, 2482, 21081, 2863, 2007, 2566, 3170, 11137, 5274, 1012, 102]

Batch input_ids shape     : torch.Size([2, 15])
Batch attention_mask shape: torch.Size([2, 15])
Vocabulary size           : 30,522
[OK] tokenizer ready.

§17.2 Tokenization and Sub-word Representations

WordPiece tokenisation (used by BERT and DistilBERT) splits rare words into frequent sub-word units, marked with ##:

"perineural"  ->  ['peri', '##neur', '##al']
"carcinoma"   ->  ['car', '##cin', '##oma']

This gives three advantages over word-level tokenisation:

  1. Fixed vocabulary (30,522 tokens for BERT-base) -- any input is representable without unknown-token collisions.
  2. Morphological sharing: "invasive", "invasion", "invasively" all share the stem invasi in the vocabulary.
  3. Biomedical robustness: clinical abbreviations and drug names that miss the full-word vocabulary are always decomposable into known sub-words.

The [CLS] embedding

After a forward pass through the model, last_hidden_state[:, 0, :] extracts the hidden state at position 0 (the [CLS] token), yielding a $d_{\text{model}}$-dimensional vector.

  • Pre-trained (not fine-tuned): generic sentence representation.
  • After fine-tuning on a task: the [CLS] vector becomes task-specific, and its geometry reflects the decision boundaries of the downstream labels.
In [3]:
# -- Feature extraction: [CLS] embeddings from the pre-trained backbone ------
backbone = AutoModel.from_pretrained(MODEL_NAME).to(device)
backbone.eval()

fe_sentences = [
    "No evidence of malignant nodule on imaging.",       # benign
    "Margins are clear with benign histology.",           # benign
    "Invasive carcinoma with lymph node involvement.",    # malignant
    "Metastatic lesion confirmed by biopsy.",             # malignant
]

with torch.no_grad():
    enc_fe = tokenizer(
        fe_sentences, padding=True, truncation=True,
        max_length=64, return_tensors="pt"
    ).to(device)
    out_fe    = backbone(**enc_fe)
    cls_embs  = out_fe.last_hidden_state[:, 0, :].cpu()  # (4, 768)

print(f"[CLS] embedding shape : {tuple(cls_embs.shape)}")

# Cosine similarity between sentence pairs
from torch.nn.functional import cosine_similarity as cos_sim
pairs = [
    (0, 1, "benign  vs benign   "),
    (0, 2, "benign  vs malignant"),
    (2, 3, "malignant vs malignant"),
]
for i, j, label in pairs:
    sim = cos_sim(cls_embs[i:i+1], cls_embs[j:j+1]).item()
    print(f"  cos({label}) = {sim:.4f}")
print("[OK] pre-trained [CLS] embeddings extracted.")
[transformers] DistilBertModel LOAD REPORT from: distilbert-base-uncased
Key                     | Status     |  | 
------------------------+------------+--+-
vocab_layer_norm.bias   | UNEXPECTED |  | 
vocab_transform.bias    | UNEXPECTED |  | 
vocab_projector.bias    | UNEXPECTED |  | 
vocab_layer_norm.weight | UNEXPECTED |  | 
vocab_transform.weight  | UNEXPECTED |  | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
[CLS] embedding shape : (4, 768)
  cos(benign  vs benign   ) = 0.8988
  cos(benign  vs malignant) = 0.9006
  cos(malignant vs malignant) = 0.9188
[OK] pre-trained [CLS] embeddings extracted.

§17.3 Fine-tuning for Sequence Classification

The pre-train / fine-tune paradigm has three steps:

  1. Pre-train once on massive unlabelled corpora (Wikipedia, BooksCorpus, PubMed). This is expensive and done once by a research lab.
  2. Download the pre-trained checkpoint: AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=k).
  3. Fine-tune on your labelled downstream task -- often a small dataset (hundreds to thousands of examples), a few epochs, and a small learning rate (~2e-5) to avoid catastrophic forgetting of pre-trained weights.
Input: [CLS]  word1  word2  ...  [SEP]
                      |
          DistilBERT (6 encoder layers)
                      |
         [CLS] hidden state  (768-dim)
                      |
            Linear(768 -> n_classes)
                      |
           Softmax -> class probabilities

Practical fine-tuning guidelines

Hyperparameter Typical range Rationale
Learning rate 1e-5 -- 5e-5 Above ~1e-4 risks catastrophic forgetting
Batch size 16 -- 32 GPU memory and gradient stability
Epochs 2 -- 5 Quick convergence; risk of overfitting on small data
Weight decay 0.01 Standard AdamW regularisation
Gradient clipping 1.0 Prevents rare large gradient spikes
In [4]:
# -- Fine-tuning DistilBERT on synthetic clinical text (3-class) -------------
import random
from torch.utils.data import Dataset, DataLoader

random.seed(42)
torch.manual_seed(42)

# Synthetic dataset -- Benign / Malignant / Uncertain
TEMPLATES = {
    0: [   # Benign / normal
        "No evidence of {t} on imaging.",
        "Margins are clear with {t} histology.",
        "Follow-up shows stable {t} without progression.",
        "The {t} appears benign with well-defined borders.",
        "No abnormal {t} detected on current workup.",
        "Resolved {t} with complete clinical response.",
    ],
    1: [   # Malignant
        "Invasive {t} with lymph node involvement.",
        "Metastatic {t} confirmed by biopsy.",
        "Aggressive {t} with perineural invasion.",
        "Malignant {t} requiring immediate intervention.",
        "High-grade {t} with poor differentiation.",
        "Recurrent {t} with extensive local infiltration.",
    ],
    2: [   # Uncertain
        "Indeterminate {t} requiring further evaluation.",
        "The {t} is inconclusive; repeat imaging recommended.",
        "Uncertain {t} -- clinical correlation advised.",
        "Pending {t} results; differential remains broad.",
        "Equivocal {t} on current workup.",
        "The {t} is atypical; additional sampling required.",
    ],
}
TERMS = ["lesion", "mass", "nodule", "opacity", "enhancement", "density", "signal"]

def make_dataset(n_per_class):
    texts, labels = [], []
    for cls, tmpls in TEMPLATES.items():
        for _ in range(n_per_class):
            texts.append(random.choice(tmpls).format(t=random.choice(TERMS)))
            labels.append(cls)
    pairs = list(zip(texts, labels))
    random.shuffle(pairs)
    return [p[0] for p in pairs], [p[1] for p in pairs]

train_texts, train_labels = make_dataset(200)   # 600 train
test_texts,  test_labels  = make_dataset(50)    # 150 test

MAX_LEN = 64
tr_enc  = tokenizer(
    train_texts, padding="max_length", truncation=True,
    max_length=MAX_LEN, return_tensors="pt"
)
te_enc  = tokenizer(
    test_texts, padding="max_length", truncation=True,
    max_length=MAX_LEN, return_tensors="pt"
)

class ClinDataset(Dataset):
    def __init__(self, enc, labs):
        self.enc  = enc
        self.labs = torch.tensor(labs, dtype=torch.long)
    def __len__(self):
        return len(self.labs)
    def __getitem__(self, i):
        return {k: v[i] for k, v in self.enc.items()}, self.labs[i]

tr_ld = DataLoader(ClinDataset(tr_enc, train_labels), batch_size=32, shuffle=True)
te_ld = DataLoader(ClinDataset(te_enc, test_labels),  batch_size=64)

# Load DistilBERT with 3-class classification head
clf = AutoModelForSequenceClassification.from_pretrained(
    MODEL_NAME, num_labels=3
).to(device)
n_p = sum(p.numel() for p in clf.parameters() if p.requires_grad)
print(f"Trainable params: {n_p:,}")

opt    = torch.optim.AdamW(clf.parameters(), lr=2e-5, weight_decay=0.01)
EPOCHS = 4
tr_losses, te_accs = [], []

for epoch in range(1, EPOCHS + 1):
    clf.train()
    ep_loss = 0.0
    for bx, by in tr_ld:
        bx = {k: v.to(device) for k, v in bx.items()}
        by = by.to(device)
        result = clf(**bx, labels=by)
        opt.zero_grad()
        result.loss.backward()
        torch.nn.utils.clip_grad_norm_(clf.parameters(), 1.0)
        opt.step()
        ep_loss += result.loss.item() * by.size(0)
    avg_loss = ep_loss / len(train_labels)

    clf.eval()
    preds, gts = [], []
    with torch.no_grad():
        for bx, by in te_ld:
            bx = {k: v.to(device) for k, v in bx.items()}
            preds.extend(clf(**bx).logits.argmax(dim=1).cpu().tolist())
            gts.extend(by.tolist())
    acc = sum(p == g for p, g in zip(preds, gts)) / len(gts)
    tr_losses.append(avg_loss)
    te_accs.append(acc)
    print(f"  Epoch {epoch}/{EPOCHS}  loss={avg_loss:.4f}  test_acc={acc:.4f}")

# Training curves
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3.5))
ax1.plot(range(1, EPOCHS + 1), tr_losses, "o-", color="steelblue", lw=2)
ax1.set_title("Training Loss"); ax1.set_xlabel("Epoch"); ax1.set_ylabel("Cross-entropy")
ax2.plot(range(1, EPOCHS + 1), [a * 100 for a in te_accs], "o-", color="darkorange", lw=2)
ax2.axhline(100 / 3, color="grey", ls="--", label="Chance (33%)")
ax2.set_title("Test Accuracy (%)"); ax2.set_xlabel("Epoch")
ax2.legend(); plt.tight_layout(); plt.show()

CLASS_NAMES = ["Benign", "Malignant", "Uncertain"]
print(classification_report(gts, preds, target_names=CLASS_NAMES))
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[transformers] DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_layer_norm.bias   | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
vocab_projector.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
vocab_transform.weight  | UNEXPECTED | 
classifier.weight       | MISSING    | 
pre_classifier.weight   | MISSING    | 
pre_classifier.bias     | MISSING    | 
classifier.bias         | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
Trainable params: 66,955,779
  Epoch 1/4  loss=0.8634  test_acc=1.0000
  Epoch 2/4  loss=0.2358  test_acc=1.0000
  Epoch 3/4  loss=0.0473  test_acc=1.0000
  Epoch 4/4  loss=0.0199  test_acc=1.0000
No description has been provided for this image
              precision    recall  f1-score   support

      Benign       1.00      1.00      1.00        50
   Malignant       1.00      1.00      1.00        50
   Uncertain       1.00      1.00      1.00        50

    accuracy                           1.00       150
   macro avg       1.00      1.00      1.00       150
weighted avg       1.00      1.00      1.00       150

§17.4 The HuggingFace pipeline API

pipeline wraps tokenisation + model forward pass + post-processing into a single callable, enabling one-line inference:

from transformers import pipeline
pipe = pipeline("text-classification",
                model="distilbert-base-uncased-finetuned-sst-2-english")
pipe("I love this!")  # -> [{'label': 'POSITIVE', 'score': 0.9998}]

The pipeline function wraps any PreTrainedModel, including a model you just fine-tuned in the current session. Requirements:

  1. Set model.config.id2label to map integer indices to readable label names.
  2. Pass model=clf, tokenizer=tokenizer to pipeline().

Available task identifiers

Task string Application
"text-classification" Sentiment, topic, clinical triage
"token-classification" NER, part-of-speech tagging
"question-answering" Extractive QA (SQuAD-style)
"zero-shot-classification" No-label-needed classification
"fill-mask" Predict masked tokens (BERT-style)
"summarization" Abstractive summarisation
In [5]:
# -- HuggingFace pipeline API (wrapping our fine-tuned classifier) -----------
CLASS_NAMES = ["Benign", "Malignant", "Uncertain"]
clf.config.id2label = {i: n for i, n in enumerate(CLASS_NAMES)}
clf.config.label2id = {n: i for i, n in enumerate(CLASS_NAMES)}

clf_pipe = hf_pipeline(
    task="text-classification",
    model=clf,
    tokenizer=tokenizer,
    device=0 if device.type == "cuda" else -1,
)

test_phrases = [
    "No evidence of malignant nodule on imaging.",
    "Invasive carcinoma with lymph node involvement.",
    "Indeterminate enhancement requiring further evaluation.",
    "Resolved opacity with complete clinical response.",
    "Equivocal density -- clinical correlation advised.",
]

print(f"{'Prediction':>12}  {'Score':>6}  Text")
print("-" * 72)
for phrase in test_phrases:
    res = clf_pipe(phrase, truncation=True, max_length=MAX_LEN)[0]
    print(f"  {res['label']:>10}  {res['score']:.4f}  {phrase}")
print()
print("[OK] pipeline inference complete.")
  Prediction   Score  Text
------------------------------------------------------------------------
      Benign  0.9878  No evidence of malignant nodule on imaging.
   Malignant  0.9883  Invasive carcinoma with lymph node involvement.
   Uncertain  0.9874  Indeterminate enhancement requiring further evaluation.
      Benign  0.9912  Resolved opacity with complete clinical response.
   Uncertain  0.9869  Equivocal density -- clinical correlation advised.

[OK] pipeline inference complete.

§17.5 Clinical NLP: BioBERT and ClinicalBERT

General-domain BERT was pre-trained on Wikipedia and BooksCorpus. Clinical text has very different statistical properties:

Property General text Clinical text
Vocabulary Everyday English Medical abbreviations, drug names, ICD codes
Sentence structure Grammatically complete Telegraphic ("No e/o mets.")
Named entities People, places, organisations Diseases, drugs, procedures, genes
Negation Rare ambiguity Critical ("no evidence of", "denies")
Sub-word fragmentation Low High (clinical terms split heavily)

BioBERT (Lee et al., 2020)

Fine-tuned BERT on PubMed abstracts (4.5 B words) and PMC full-text articles (13.5 B words). State-of-the-art on biomedical NER, relation extraction, and QA tasks.

HuggingFace Hub: dmis-lab/biobert-base-cased-v1.1

ClinicalBERT (Alsentzer et al., 2019)

Continued pre-training of BioBERT on ~2 M MIMIC-III clinical notes (ICU records from Beth Israel Deaconess Medical Center). Captures clinical abbreviations, lab values, and nursing narrative.

HuggingFace Hub: emilyalsentzer/Bio_ClinicalBERT

Domain adaptation strategy

General BERT  ---(continue pre-train on PubMed)---> BioBERT
BioBERT       ---(continue pre-train on MIMIC)----> ClinicalBERT
ClinicalBERT  ---(fine-tune on task labels)-------> Task model

Each arrow applies the same Transformer encoder architecture and masked language modelling objective; only the pre-training corpus changes.

In [6]:
# -- PCA visualisation of fine-tuned [CLS] embeddings (test set) -------------
clf.eval()
all_embs, all_labs = [], []

with torch.no_grad():
    for bx, by in te_ld:
        bx    = {k: v.to(device) for k, v in bx.items()}
        # Access the DistilBERT backbone directly to get last_hidden_state
        out_v = clf.distilbert(**bx)
        cls_v = out_v.last_hidden_state[:, 0, :].cpu().numpy()
        all_embs.append(cls_v)
        all_labs.extend(by.tolist())

all_embs = np.concatenate(all_embs, axis=0)   # (150, 768)
all_labs  = np.array(all_labs)

pca   = PCA(n_components=2, random_state=42)
emb2d = pca.fit_transform(all_embs)

var1, var2 = pca.explained_variance_ratio_
print(f"PCA explained variance: PC1={var1:.3f}  PC2={var2:.3f}  "
      f"total={var1 + var2:.3f}")

fig, ax = plt.subplots(figsize=(7, 5.5))
colors = ["steelblue", "firebrick", "darkorange"]
for cls_idx, (cname, col) in enumerate(zip(CLASS_NAMES, colors)):
    mask = all_labs == cls_idx
    ax.scatter(emb2d[mask, 0], emb2d[mask, 1], c=col, label=cname,
               alpha=0.75, edgecolors="white", linewidths=0.4, s=60)
ax.set_title("PCA of Fine-tuned [CLS] Embeddings (test set)", fontsize=13)
ax.set_xlabel(f"PC1 ({var1:.1%} var.)")
ax.set_ylabel(f"PC2 ({var2:.1%} var.)")
ax.legend(title="Class", framealpha=0.9)
plt.tight_layout(); plt.show()
print("[OK] PCA embedding visualisation complete.")
PCA explained variance: PC1=0.512  PC2=0.472  total=0.984

No description has been provided for this image
[OK] PCA embedding visualisation complete.

§17.6 Summary

Key Concepts

Concept Description
WordPiece tokenisation Sub-word vocabulary (30,522 tokens); any word decomposable
MLM pre-training Predict 15% masked tokens from full bidirectional context
[CLS] embedding Sentence-level representation; task-specific after fine-tuning
Fine-tuning Update all weights with small LR on labelled downstream data
pipeline API One-call inference: tokenise + forward pass + label decode
Domain adaptation Continue MLM pre-training on target-domain text corpus
BioBERT / ClinicalBERT Domain-adapted checkpoints for PubMed / ICU clinical notes

Connections to Earlier Chapters

Chapter Connection
Ch07 (Logistic Regression) Linear classification head on top of [CLS] embedding
Ch06 / Ch14 (Regularisation) Weight decay in AdamW prevents fine-tuning overfitting
Ch14 (Neural Networks) Multi-layer stacked encoder blocks, cross-entropy loss
Ch15 (CNNs) Transfer learning / fine-tuning paradigm is identical
Ch16 (Transformers) BERT IS the Transformer encoder built in Ch16

What's Next?

Chapter 18 will explore Vision Transformers (ViT) -- applying the identical attention mechanism to image patches -- and multimodal models that combine image and text representations for tasks such as medical imaging report generation.


Section 17.7 — Summary & Quick Reference

Key Formulas

Concept Formula Python
MLM pre-training loss $\mathcal{L}_{\text{MLM}} = -\!\sum_{m \in \mathcal{M}} \log P(x_m \mid x_{\backslash m};\,\theta)$ AutoModelForMaskedLM
Fine-tuning CE loss $\mathcal{L} = -\!\sum_i y_i \log \hat{p}_i$ nn.CrossEntropyLoss()
[CLS] classification head $\hat{\mathbf{p}} = \text{softmax}(\mathbf{W}\mathbf{h}_{[\text{CLS}]} + \mathbf{b})$ AutoModelForSequenceClassification
AdamW update $\theta \leftarrow \theta(1-\eta\lambda) - \eta\,\hat{\mathbf{m}}/(\sqrt{\hat{\mathbf{v}}}+\varepsilon)$ AdamW(lr=2e-5, weight_decay=0.01)
WordPiece vocab 30,522 tokens (BERT-base); sub-word decomposition AutoTokenizer.from_pretrained(...)

ML Connections

Concept Where It Appears
Pre-training / fine-tuning Same paradigm as CNNs in Ch15; large-corpus initialisation, then task-specific adaptation
[CLS] embedding Sentence-level feature vector; input to any downstream classifier (Ch07, Ch14)
Weight decay (AdamW) L2 regularisation on weights (Ch06, Ch14); prevents overfitting on small fine-tuning sets
Cross-entropy loss Same loss used in every classifier from Ch07 onward and in Ch14 neural networks
Domain adaptation BioBERT, ClinicalBERT -- continue MLM pre-training on target corpus; same math, new data

What's Next?

$$\boxed{\text{Ch 17: BERT}} \;\to\; \boxed{\text{Ch 18: Vision Transformers}} \;\to\; \boxed{\text{Ch 19: Graph Neural Networks}} \;\to\; \boxed{\text{Ch 20: Cognitive Agent}}$$

Chapter 18 applies the identical Transformer encoder to images: each image is divided into fixed-size patches, each patch is projected to a $d_{\text{model}}$-dimensional embedding, and the resulting sequence is processed by the same multi-head attention blocks built in Chapter 16. The only difference is what gets tokenised.

References

  1. Devlin J, Chang M-W, Lee K, Toutanova K (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT, 4171-4186.
  2. Sanh V, Debut L, Chaumond J, Wolf T (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108.
  3. Lee J, Yoon W, Kim S, et al. (2020). BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4), 1234-1240.
  4. Alsentzer E, Murphy J, Boag W, et al. (2019). Publicly available clinical BERT embeddings. NAACL Clinical NLP Workshop, 72-78.
  5. Vaswani A, Shazeer N, Parmar N, et al. (2017). Attention is all you need. NeurIPS, 30.
  6. Liu Y, Ott M, Goyal N, et al. (2019). RoBERTa: a robustly optimized BERT pretraining approach. arXiv:1907.11692.
  7. Wolf T, Debut L, Sanh V, et al. (2020). HuggingFace's Transformers: state-of-the-art natural language processing. EMNLP Systems, 38-45.
  8. Loshchilov I, Hutter F (2019). Decoupled weight decay regularisation. ICLR.
  9. Johnson AEW, Pollard TJ, Shen L, et al. (2016). MIMIC-III, a freely accessible critical care database. Scientific Data, 3, 160035.
  10. Goodfellow I, Bengio Y, Courville A (2016). Deep Learning. MIT Press.