Statistical Basis of AI/ML¶
Graph Neural Networks¶
Chapter 19 -- Message Passing, Node Classification, and Molecular Property Prediction¶
Chapters 16-18 applied self-attention to sequences and images. This chapter introduces the third major structural inductive bias in deep learning: the graph. We build a Graph Convolutional Network (GCN) entirely from first principles using PyTorch, train it on node classification and graph classification tasks, and apply it to molecular property prediction.
Why This Chapter?¶
Many real-world systems are naturally represented as graphs: molecules (atoms + bonds), protein interaction networks, citation graphs, medical knowledge graphs, and social networks. Standard neural architectures assume:
- Sequences (Transformers): fixed-order tokens.
- Grids (CNNs): regular pixel lattices.
Graphs break both assumptions: nodes have no canonical order, and each node may have a different number of neighbours. Graph Neural Networks (GNNs) address this with a permutation-invariant message passing framework that generalises convolution to arbitrary topologies.
| Section | Topic |
|---|---|
| §19.0 | Setup and library overview |
| §19.1 | Graph fundamentals: adjacency, degree, and Laplacian |
| §19.2 | Graph Convolutional Networks (GCN): message passing theory |
| §19.3 | Semi-supervised node classification on a synthetic community graph |
| §19.4 | Graph-level classification with global readout pooling |
| §19.5 | Molecular property prediction: atoms as nodes, bonds as edges |
| §19.6 | Biological and medical applications |
| §19.7 | Summary and references |
§19.0 Setup¶
| Library | Role in this chapter |
|---|---|
torch / torch.nn |
GCNLayer, GCN model, training loops |
numpy |
Adjacency matrix construction and normalisation |
matplotlib |
Graph visualisation, training curves, attention maps |
seaborn |
Heatmaps and styled plots |
sklearn |
PCA for embedding visualisation, classification metrics |
No external graph library (PyTorch Geometric, DGL) is required. The GCN is implemented directly using dense matrix multiplication, which is sufficient for the small graphs used in this chapter.
# -- Imports ------------------------------------------------------------------
import warnings
warnings.filterwarnings("ignore")
import random
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.metrics import classification_report
from sklearn.decomposition import PCA
sns.set_theme(style="whitegrid", font_scale=1.05)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"PyTorch : {torch.__version__}")
print(f"Device : {device}")
if device.type == "cuda":
print(f"GPU : {torch.cuda.get_device_name(0)}")
vram = torch.cuda.get_device_properties(0).total_memory / 1e9
print(f"VRAM : {vram:.1f} GB")
print("Chapter 19 -- Graph Neural Networks: environment ready.")
§19.1 Graph Fundamentals¶
A graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ consists of:
- A node set $\mathcal{V}$ with $|\mathcal{V}| = N$ nodes.
- An edge set $\mathcal{E} \subseteq \mathcal{V} \times \mathcal{V}$.
Matrix representations¶
Adjacency matrix $\mathbf{A} \in \{0,1\}^{N \times N}$: $$A_{ij} = 1 \iff (i, j) \in \mathcal{E}$$
Degree matrix $\mathbf{D}$ (diagonal): $D_{ii} = \sum_j A_{ij}$ (out-degree of node $i$).
Graph Laplacian: $\mathbf{L} = \mathbf{D} - \mathbf{A}$.
Normalised Laplacian: $\mathbf{L}_{\text{sym}} = \mathbf{I} - \mathbf{D}^{-1/2}\mathbf{A}\mathbf{D}^{-1/2}$.
The eigenvalues of $\mathbf{L}_{\text{sym}}$ lie in $[0, 2]$ and describe the graph's frequency spectrum. Spectral GNNs (Bruna et al., 2014) define convolution in this frequency domain; GCN (Kipf and Welling, 2017) approximates this with a first-order Chebyshev polynomial, yielding the simple message-passing rule we implement in §19.2.
Stochastic Block Model (SBM)¶
The Stochastic Block Model generates random graphs with community structure:
- Nodes are partitioned into $K$ communities.
- Edge probability within a community: $p_{\text{intra}}$.
- Edge probability between communities: $p_{\text{inter}} \ll p_{\text{intra}}$.
SBM is the standard benchmark for node classification because the ground-truth community label is perfectly determined by graph topology alone.
# -- Build a Stochastic Block Model (SBM) graph ------------------------------
rng = np.random.default_rng(42)
N_NODES = 60
N_COMM = 4
COMM_SIZE = N_NODES // N_COMM # 15 nodes per community
P_INTRA = 0.55
P_INTER = 0.04
adj = np.zeros((N_NODES, N_NODES), dtype=np.float32)
for i in range(N_NODES):
for j in range(i + 1, N_NODES):
ci, cj = i // COMM_SIZE, j // COMM_SIZE
p = P_INTRA if ci == cj else P_INTER
if rng.random() < p:
adj[i, j] = adj[j, i] = 1.0
# Node positions: communities in quadrants with Gaussian jitter
quadrant_centers = [(-2.5, 2.5), (2.5, 2.5), (-2.5, -2.5), (2.5, -2.5)]
positions = np.zeros((N_NODES, 2))
for c in range(N_COMM):
cx, cy = quadrant_centers[c]
start = c * COMM_SIZE
for k in range(COMM_SIZE):
positions[start + k] = [cx + rng.standard_normal() * 0.55,
cy + rng.standard_normal() * 0.55]
COMM_COLORS = ["tab:red", "tab:green", "tab:blue", "tab:orange"]
node_colors = [COMM_COLORS[i // COMM_SIZE] for i in range(N_NODES)]
fig, ax = plt.subplots(figsize=(7, 6.5))
# Draw edges first (low zorder)
for i in range(N_NODES):
for j in range(i + 1, N_NODES):
if adj[i, j]:
ax.plot(
[positions[i, 0], positions[j, 0]],
[positions[i, 1], positions[j, 1]],
color="black", alpha=0.12, lw=0.8, zorder=1,
)
# Draw nodes
for i in range(N_NODES):
ax.scatter(positions[i, 0], positions[i, 1],
c=node_colors[i], s=130, zorder=2,
edgecolors="white", linewidths=0.9)
# Legend
from matplotlib.lines import Line2D
legend_handles = [
Line2D([0], [0], marker="o", color="w", markerfacecolor=COMM_COLORS[c],
markersize=10, label=f"Community {c}")
for c in range(N_COMM)
]
ax.legend(handles=legend_handles, loc="upper center", ncol=4,
bbox_to_anchor=(0.5, -0.04), frameon=False)
ax.set_title(
f"Stochastic Block Model Graph ({N_NODES} nodes, {N_COMM} communities)\n"
f"p_intra={P_INTRA}, p_inter={P_INTER}",
fontsize=11,
)
ax.axis("off")
plt.tight_layout()
plt.show()
n_edges = int(adj.sum()) // 2
avg_deg = adj.sum(1).mean()
print(f"Nodes : {N_NODES}")
print(f"Edges : {n_edges}")
print(f"Communities : {N_COMM}")
print(f"Avg degree : {avg_deg:.1f}")
print("[OK] graph construction complete.")
§19.2 Graph Convolutional Networks: Message Passing¶
The key idea¶
Standard convolution slides a fixed kernel over a regular grid. On a graph, the "grid" varies per node (different degree). GCN replaces the kernel with a neighbourhood aggregation step:
Each node collects feature vectors from its neighbours, averages them, and applies a shared linear transformation.
Formally (Kipf and Welling, 2017):
$$\mathbf{H}^{(l+1)} = \sigma\!\left(\tilde{\mathbf{D}}^{-1/2} \tilde{\mathbf{A}} \tilde{\mathbf{D}}^{-1/2} \mathbf{H}^{(l)} \mathbf{W}^{(l)}\right)$$
where:
| Symbol | Meaning |
|---|---|
| $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$ | Adjacency with self-loops added |
| $\tilde{\mathbf{D}}_{ii} = \sum_j \tilde{A}_{ij}$ | Degree matrix of $\tilde{\mathbf{A}}$ |
| $\tilde{\mathbf{D}}^{-1/2} \tilde{\mathbf{A}} \tilde{\mathbf{D}}^{-1/2}$ | Symmetric normalised adjacency $\hat{\mathbf{A}}$ |
| $\mathbf{H}^{(l)} \in \mathbb{R}^{N \times d_l}$ | Node features at layer $l$ |
| $\mathbf{W}^{(l)} \in \mathbb{R}^{d_l \times d_{l+1}}$ | Learnable weight matrix |
| $\sigma$ | Non-linear activation (ReLU for hidden layers) |
Message passing interpretation¶
For each node $i$, the update is:
$$\mathbf{h}_i^{(l+1)} = \sigma\!\left(\mathbf{W}^{(l)} \cdot \frac{1}{\sqrt{d_i d_j}} \sum_{j \in \mathcal{N}(i) \cup \{i\}} \mathbf{h}_j^{(l)}\right)$$
- Aggregate: collect scaled neighbour features (messages).
- Update: apply linear transform and activation.
- Self-loop ($\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$): node always includes its own features in the aggregation.
With $L$ layers, each node integrates information from its $L$-hop neighbourhood. For node classification, $L = 2$ is standard; larger $L$ causes over-smoothing (all node representations converge to the same vector).
# -- GCNLayer, GCN, and normalised adjacency helper --------------------------
def norm_adj(adj_np):
# A_hat = A + I (add self-loops)
a_hat = adj_np + np.eye(len(adj_np), dtype=np.float32)
d = a_hat.sum(1)
d_inv_sqrt = np.diag((d + 1e-8) ** -0.5)
return (d_inv_sqrt @ a_hat @ d_inv_sqrt).astype(np.float32)
class GCNLayer(nn.Module):
def __init__(self, in_dim, out_dim):
super().__init__()
self.W = nn.Linear(in_dim, out_dim, bias=False)
nn.init.xavier_uniform_(self.W.weight)
def forward(self, x, a_norm):
# a_norm: (N, N), x: (N, in_dim) -> (N, out_dim)
return a_norm @ self.W(x)
class GCN(nn.Module):
def __init__(self, in_dim, hidden_dim, out_dim, n_layers=2):
super().__init__()
dims = [in_dim] + [hidden_dim] * (n_layers - 1) + [out_dim]
self.layers = nn.ModuleList(
[GCNLayer(dims[i], dims[i + 1]) for i in range(n_layers)]
)
def forward(self, x, a_norm, return_hidden=False):
for layer in self.layers[:-1]:
x = F.relu(layer(x, a_norm))
h = x # penultimate embedding
out = self.layers[-1](h, a_norm) # final layer (no activation)
if return_hidden:
return out, h
return out
# Precompute normalised adjacency for SBM graph
A_norm = torch.tensor(norm_adj(adj)).to(device)
# Verify architecture
gcn_test = GCN(in_dim=16, hidden_dim=64, out_dim=4, n_layers=3).to(device)
x_dummy = torch.randn(N_NODES, 16).to(device)
out_test = gcn_test(x_dummy, A_norm)
print(f"A_norm shape : {tuple(A_norm.shape)}")
print(f"Input shape : {tuple(x_dummy.shape)}")
print(f"GCN output shape : {tuple(out_test.shape)}")
print("[OK] GCN architecture built.")
§19.3 Semi-Supervised Node Classification¶
Setting¶
The classic GCN paper (Kipf and Welling, 2017) uses a transductive semi-supervised setting:
- All $N$ nodes participate in message passing (the full graph is seen).
- Only a small fraction of nodes have ground-truth labels for training.
- The model propagates label information through the graph structure.
In our SBM experiment:
| Split | Count | Notes |
|---|---|---|
| Labelled (train) | 20 | 5 per community |
| Unlabelled (test) | 40 | evaluated post-training |
Why does GCN work here?¶
With only 5 labelled nodes per community, the GCN propagates label information through the graph topology. After 2-3 layers:
- Layer 1: each node aggregates its 1-hop neighbourhood.
- Layer 2: each node aggregates its 2-hop neighbourhood.
- Intra-community edges ($p = 0.55$) carry consistent label signals; the few inter-community edges ($p = 0.04$) introduce minor noise.
The learned embeddings should form 4 distinct clusters -- one per community -- even though only 1/3 of the data has labels.
# -- Semi-supervised node classification on SBM ------------------------------
torch.manual_seed(42)
rng = np.random.default_rng(42)
N_FEAT = 16
X = torch.randn(N_NODES, N_FEAT).to(device)
y = torch.tensor([i // COMM_SIZE for i in range(N_NODES)],
dtype=torch.long).to(device)
# Train mask: 5 labelled nodes per community (first 5 in each block)
LABELLED_PER_COMM = 5
train_mask = torch.zeros(N_NODES, dtype=torch.bool)
for c in range(N_COMM):
start = c * COMM_SIZE
train_mask[start : start + LABELLED_PER_COMM] = True
test_mask = ~train_mask
print(f"Train nodes : {train_mask.sum().item()} ({N_COMM} x {LABELLED_PER_COMM})")
print(f"Test nodes : {test_mask.sum().item()}")
gcn_nc = GCN(in_dim=N_FEAT, hidden_dim=64, out_dim=N_COMM, n_layers=3).to(device)
opt_nc = torch.optim.Adam(gcn_nc.parameters(), lr=8e-3, weight_decay=5e-4)
EPOCHS_NC = 150
tr_losses_nc, te_accs_nc = [], []
for epoch in range(1, EPOCHS_NC + 1):
gcn_nc.train()
logits = gcn_nc(X, A_norm)
loss = F.cross_entropy(logits[train_mask], y[train_mask])
opt_nc.zero_grad()
loss.backward()
opt_nc.step()
preds_test = logits[test_mask].argmax(1).detach()
acc = (preds_test == y[test_mask]).float().mean().item()
tr_losses_nc.append(loss.item())
te_accs_nc.append(acc)
if epoch % 30 == 0:
print(f" Epoch {epoch:3d} loss={loss.item():.4f} test_acc={acc:.4f}")
# Training curves
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3.5))
ax1.plot(range(1, EPOCHS_NC + 1), tr_losses_nc, color="steelblue", lw=2)
ax1.set_title("Training Loss (labelled nodes)")
ax1.set_xlabel("Epoch")
ax1.set_ylabel("Cross-entropy")
ax2.plot(range(1, EPOCHS_NC + 1), [a * 100 for a in te_accs_nc],
color="darkorange", lw=2)
ax2.axhline(25, color="grey", ls="--", label="Chance (25%)")
ax2.set_title("Test Accuracy (%)")
ax2.set_xlabel("Epoch")
ax2.legend()
plt.tight_layout()
plt.show()
# PCA of learned embeddings
gcn_nc.eval()
with torch.no_grad():
_, hidden = gcn_nc(X, A_norm, return_hidden=True)
embeds = hidden.cpu().numpy()
pca = PCA(n_components=2)
emb2d = pca.fit_transform(embeds)
fig, ax = plt.subplots(figsize=(6, 5))
for c in range(N_COMM):
mask_c = np.array([i // COMM_SIZE == c for i in range(N_NODES)])
marker = "D" if np.any(train_mask.cpu().numpy() & mask_c) else "o"
ax.scatter(emb2d[mask_c, 0], emb2d[mask_c, 1],
c=COMM_COLORS[c], s=90, label=f"Comm {c}",
edgecolors="white", linewidths=0.7)
ax.set_title("PCA of GCN Node Embeddings (penultimate layer)\n"
"Colour = ground-truth community")
ax.set_xlabel("PC 1")
ax.set_ylabel("PC 2")
ax.legend(loc="best", fontsize=9)
plt.tight_layout()
plt.show()
# Classification report on test nodes
preds_np = logits[test_mask].argmax(1).detach().cpu().numpy()
true_np = y[test_mask].cpu().numpy()
print(classification_report(true_np, preds_np,
target_names=[f"Comm {c}" for c in range(N_COMM)]))
§19.4 Graph-Level Classification and Readout Pooling¶
From node embeddings to graph embeddings¶
Node classification produces one label per node. Graph classification requires a single vector representing the entire graph. This is achieved via a readout (pooling) function applied to all node embeddings after the final GCN layer:
| Readout | Formula | Notes |
|---|---|---|
| Mean | $\mathbf{h}_G = \frac{1}{N} \sum_i \mathbf{h}_i$ | Insensitive to graph size |
| Sum | $\mathbf{h}_G = \sum_i \mathbf{h}_i$ | Counts features; size-sensitive |
| Max | $\mathbf{h}_G = \max_i \mathbf{h}_i$ | Captures extreme features |
Mean pooling is the standard baseline; sum pooling is used when absolute feature counts matter (e.g., molecular formula); max pooling highlights rare but discriminating features.
Graph classification pipeline¶
Input batch of graphs {G_1, G_2, ..., G_B}
|
For each G_k:
GCN layers: X_k -> H_k (N_k x hidden_dim)
Mean pool: h_k = mean(H_k, dim=0) (hidden_dim,)
|
Stack: [h_1, h_2, ..., h_B] (B x hidden_dim)
|
Linear head -> logits (B x n_classes)
Synthetic task¶
We generate 200 random Erdos-Renyi graphs with 4-12 nodes each. The binary label is:
$$y = \begin{cases} 1 & \text{if } N > 8 \\ 0 & \text{otherwise} \end{cases}$$
The GCN must learn, through graph structure alone, whether a graph is "large" or "small" -- a task that requires counting-like aggregation across all nodes.
# -- Graph-level binary classification on synthetic ER graphs ----------------
random.seed(42)
rng = np.random.default_rng(42)
torch.manual_seed(42)
N_FEAT_G = 8 # node feature dimension
N_GRAPHS = 200
THRESHOLD = 8 # label=1 if n_nodes > THRESHOLD
def make_er_graph(n_nodes, p_edge=0.35, n_feat=N_FEAT_G):
# Build ER adjacency + enforce connectivity via chain
a = np.zeros((n_nodes, n_nodes), dtype=np.float32)
for i in range(n_nodes):
for j in range(i + 1, n_nodes):
if rng.random() < p_edge:
a[i, j] = a[j, i] = 1.0
for i in range(n_nodes - 1): # chain ensures connectivity
a[i, i + 1] = a[i + 1, i] = 1.0
x = rng.standard_normal((n_nodes, n_feat)).astype(np.float32)
return x, a
# Generate dataset
graphs_gc = []
for _ in range(N_GRAPHS):
n = rng.integers(4, 13) # 4-12 nodes
x, a = make_er_graph(n)
label = 1 if n > THRESHOLD else 0
graphs_gc.append((x, a, label))
random.shuffle(graphs_gc)
n_train_g = 160
tr_gc = graphs_gc[:n_train_g]
te_gc = graphs_gc[n_train_g:]
labels_tr = [g[2] for g in tr_gc]
labels_te = [g[2] for g in te_gc]
print(f"Train: {n_train_g} graphs (label=1: {sum(labels_tr)}, label=0: {n_train_g - sum(labels_tr)})")
print(f"Test: {len(te_gc)} graphs (label=1: {sum(labels_te)}, label=0: {len(te_gc) - sum(labels_te)})")
class GraphGCN(nn.Module):
def __init__(self, in_dim, hidden_dim, n_classes):
super().__init__()
self.conv1 = GCNLayer(in_dim, hidden_dim)
self.conv2 = GCNLayer(hidden_dim, hidden_dim)
self.head = nn.Linear(hidden_dim, n_classes)
def forward(self, x, a_norm):
h = F.relu(self.conv1(x, a_norm))
h = F.relu(self.conv2(h, a_norm))
g = h.mean(0, keepdim=True) # mean readout: (1, hidden_dim)
return self.head(g) # (1, n_classes)
graph_gcn = GraphGCN(in_dim=N_FEAT_G, hidden_dim=32, n_classes=2).to(device)
opt_gc = torch.optim.Adam(graph_gcn.parameters(), lr=5e-3, weight_decay=1e-4)
sch_gc = torch.optim.lr_scheduler.StepLR(opt_gc, step_size=10, gamma=0.7)
EPOCHS_GC = 30
gc_tr_losses, gc_te_accs = [], []
for epoch in range(1, EPOCHS_GC + 1):
graph_gcn.train()
ep_loss = 0.0
random.shuffle(tr_gc)
for x_g, a_g, lab_g in tr_gc:
xt = torch.tensor(x_g).to(device)
at = torch.tensor(norm_adj(a_g)).to(device)
lab = torch.tensor([lab_g], dtype=torch.long).to(device)
out = graph_gcn(xt, at)
loss = F.cross_entropy(out, lab)
opt_gc.zero_grad()
loss.backward()
opt_gc.step()
ep_loss += loss.item()
sch_gc.step()
avg_loss = ep_loss / len(tr_gc)
graph_gcn.eval()
te_preds, te_true = [], []
with torch.no_grad():
for x_g, a_g, lab_g in te_gc:
xt = torch.tensor(x_g).to(device)
at = torch.tensor(norm_adj(a_g)).to(device)
pred = graph_gcn(xt, at).argmax(1).item()
te_preds.append(pred)
te_true.append(lab_g)
acc = sum(p == t for p, t in zip(te_preds, te_true)) / len(te_gc)
gc_tr_losses.append(avg_loss)
gc_te_accs.append(acc)
if epoch % 5 == 0:
print(f" Epoch {epoch:2d}/{EPOCHS_GC} loss={avg_loss:.4f} test_acc={acc:.4f}")
# Training curves
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3.5))
ax1.plot(range(1, EPOCHS_GC + 1), gc_tr_losses, "o-", color="steelblue", lw=2, ms=4)
ax1.set_title("Graph Classification -- Training Loss")
ax1.set_xlabel("Epoch")
ax1.set_ylabel("Cross-entropy")
ax2.plot(range(1, EPOCHS_GC + 1), [a * 100 for a in gc_te_accs],
"o-", color="darkorange", lw=2, ms=4)
ax2.axhline(50, color="grey", ls="--", label="Chance (50%)")
ax2.set_title("Test Accuracy (%)")
ax2.set_xlabel("Epoch")
ax2.legend()
plt.tight_layout()
plt.show()
print(classification_report(te_true, te_preds,
target_names=["Small (<=8)", "Large (>8)"]))
print("[OK] graph classification complete.")
§19.5 Molecular Property Prediction¶
Atoms as nodes, bonds as edges¶
A molecule maps naturally to a graph:
- Nodes: atoms with feature vectors encoding atom type, formal charge, hybridisation, and degree.
- Edges: covalent bonds (adjacency matrix); may be typed (single, double, aromatic) by encoding edge features separately.
Feature encoding¶
| Feature | Encoding |
|---|---|
| Atom type (C, O, N, ...) | One-hot over vocabulary |
| Atomic number | Integer or normalised float |
| Degree (n bonds) | Integer |
| Aromaticity | Binary |
For this demonstration we use a minimal encoding:
- 3 atom types: C (carbon), O (oxygen), N (nitrogen)
- One-hot feature vector of length 3
- Chain / branched topology (no ring structures)
- Binary label: does the molecule contain at least one oxygen atom?
GCN + mean pooling for molecular graphs¶
The same GraphGCN from §19.4 applies directly. The only change is the
feature dimension (3 instead of 8) and the data generation procedure.
Scale: real molecular datasets¶
| Dataset | Molecules | Task |
|---|---|---|
| QM9 | 134 K | 19 quantum properties (regression) |
| ESOL | 1 K | Water solubility (regression) |
| HIV | 41 K | HIV replication inhibition (binary) |
| PCBA | 437 K | 128 bioassay endpoints (multi-task) |
State-of-the-art models (Hu et al., 2020 -- Strategies for Pre-training GNNs) pre-train on millions of unlabelled molecules and fine-tune on downstream tasks, mirroring the pre-train / fine-tune paradigm of BERT (Chapter 17).
# -- Molecular property prediction: chain molecules, binary has-oxygen label -
random.seed(7)
rng = np.random.default_rng(7)
torch.manual_seed(7)
# Atom types: 0=C, 1=O, 2=N
ATOM_VOCAB = {0: "C", 1: "O", 2: "N"}
N_ATOM_TYPES = len(ATOM_VOCAB)
def make_molecule(n_carbon, n_oxygen, n_nitrogen):
# Concatenate atoms in order, connect as a chain
atoms = [0] * n_carbon + [1] * n_oxygen + [2] * n_nitrogen
n = len(atoms)
a = np.zeros((n, n), dtype=np.float32)
for i in range(n - 1):
a[i, i + 1] = a[i + 1, i] = 1.0
x = np.eye(N_ATOM_TYPES, dtype=np.float32)[atoms] # one-hot (n, 3)
label = 1 if n_oxygen > 0 else 0
return x, a, label
molecules_all = []
# Pure carbon chains (label 0)
for nc in range(1, 9):
for _ in range(12):
molecules_all.append(make_molecule(nc, 0, 0))
# Oxygen-containing chains (label 1)
for nc in range(1, 7):
for no in range(1, 4):
molecules_all.append(make_molecule(nc, no, 0))
for ni in range(1, 3):
molecules_all.append(make_molecule(nc, 1, ni))
random.shuffle(molecules_all)
n_mol = len(molecules_all)
n_tr = int(0.8 * n_mol)
mol_tr = molecules_all[:n_tr]
mol_te = molecules_all[n_tr:]
labs_tr = [m[2] for m in mol_tr]
labs_te = [m[2] for m in mol_te]
print(f"Molecules total : {n_mol}")
print(f"Train : {n_tr} (label=1: {sum(labs_tr)}, label=0: {n_tr - sum(labs_tr)})")
print(f"Test : {n_mol - n_tr} (label=1: {sum(labs_te)}, label=0: {n_mol - n_tr - sum(labs_te)})")
mol_gcn = GraphGCN(in_dim=N_ATOM_TYPES, hidden_dim=32, n_classes=2).to(device)
opt_mol = torch.optim.Adam(mol_gcn.parameters(), lr=5e-3, weight_decay=1e-4)
EPOCHS_MOL = 40
mol_tr_losses, mol_te_accs = [], []
for epoch in range(1, EPOCHS_MOL + 1):
mol_gcn.train()
ep_loss = 0.0
random.shuffle(mol_tr)
for xm, am, ym in mol_tr:
xt = torch.tensor(xm).to(device)
at = torch.tensor(norm_adj(am)).to(device)
lab = torch.tensor([ym], dtype=torch.long).to(device)
out = mol_gcn(xt, at)
loss = F.cross_entropy(out, lab)
opt_mol.zero_grad()
loss.backward()
opt_mol.step()
ep_loss += loss.item()
avg_loss = ep_loss / len(mol_tr)
mol_gcn.eval()
mol_preds, mol_true = [], []
with torch.no_grad():
for xm, am, ym in mol_te:
xt = torch.tensor(xm).to(device)
at = torch.tensor(norm_adj(am)).to(device)
p = mol_gcn(xt, at).argmax(1).item()
mol_preds.append(p)
mol_true.append(ym)
acc = sum(p == t for p, t in zip(mol_preds, mol_true)) / len(mol_te)
mol_tr_losses.append(avg_loss)
mol_te_accs.append(acc)
if epoch % 10 == 0:
print(f" Epoch {epoch:2d}/{EPOCHS_MOL} loss={avg_loss:.4f} test_acc={acc:.4f}")
# Training curves
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3.5))
ax1.plot(range(1, EPOCHS_MOL + 1), mol_tr_losses, "o-", color="steelblue", lw=2, ms=4)
ax1.set_title("Molecular GCN -- Training Loss")
ax1.set_xlabel("Epoch")
ax1.set_ylabel("Cross-entropy")
ax2.plot(range(1, EPOCHS_MOL + 1), [a * 100 for a in mol_te_accs],
"o-", color="purple", lw=2, ms=4)
ax2.axhline(50, color="grey", ls="--", label="Chance (50%)")
ax2.set_title("Has-Oxygen Classification Accuracy (%)")
ax2.set_xlabel("Epoch")
ax2.legend()
plt.tight_layout()
plt.show()
print(classification_report(mol_true, mol_preds,
target_names=["No oxygen", "Has oxygen"]))
print("[OK] molecular property prediction complete.")
§19.6 Biological and Medical Applications¶
Drug discovery: molecular property prediction¶
GNNs have become the standard architecture for ADMET prediction (Absorption, Distribution, Metabolism, Excretion, Toxicity):
- Atom-level GCN: Duvenaud et al. (2015) replaced fixed-radius Morgan fingerprints with learned atom embeddings. The graph convolution acts as a differentiable fingerprint.
- MPNN (Gilmer et al., 2017): unified framework generalising GCN, GGNN, and spectral approaches. Achieves DFT-level accuracy on QM9 properties.
- Pre-training strategies (Hu et al., 2020): mask atom types during pre-training (analogous to BERT masked LM), fine-tune on downstream bioassays.
Protein structure: graph-of-residues¶
A protein's 3D structure can be encoded as a k-nearest-neighbour graph of amino acid residues:
- Nodes: residues with features (amino acid type, secondary structure, solvent accessibility).
- Edges: spatial proximity (Euclidean distance < 8 A) or sequence adjacency.
- Task: function prediction, binding site detection, variant pathogenicity.
AlphaFold 2 (Jumper et al., 2021) uses a paired attention mechanism on the multiple-sequence alignment (Evoformer block) that can be interpreted as a GNN over the co-evolution graph.
Medical knowledge graphs¶
Knowledge graph (KG) completion predicts missing (entity, relation, entity) triples in curated biomedical ontologies:
| KG | Nodes | Relations | Application |
|---|---|---|---|
| DrugBank | 10 K+ | 30+ | Drug-drug interaction prediction |
| UMLS | 3.5 M | 900+ | Clinical concept disambiguation |
| PrimeKG | 130 K | 30 | Therapeutic target prediction |
RGCN (Schlichtkrull et al., 2018) extends GCN to relation-typed graphs, using a separate weight matrix per relation type.
Radiology report graphs¶
Recent work encodes clinical reports as scene graphs (entities + relations) and uses GNNs to reason over anatomical findings -- connecting the NLP models of Chapter 17 with the image models of Chapter 18 via a unified graph representation.
§19.7 Summary¶
Key Concepts¶
| Concept | Description |
|---|---|
| Graph | $\mathcal{G} = (\mathcal{V}, \mathcal{E})$; nodes + edges; no fixed order or dimensionality |
| Normalised adjacency | $\hat{\mathbf{A}} = \tilde{\mathbf{D}}^{-1/2}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-1/2}$; self-loops added; spectral-stable |
| GCN layer | $\mathbf{H}' = \sigma(\hat{\mathbf{A}}\mathbf{H}\mathbf{W})$; shared linear weight over all nodes |
| Message passing | Each node aggregates neighbour features; $L$ layers = $L$-hop receptive field |
| Over-smoothing | Too many GCN layers collapse all node embeddings to the same vector; typical $L = 2$-$3$ |
| Readout pooling | Mean/sum/max over node embeddings yields a fixed-size graph embedding |
| Molecular graph | Atoms = nodes (one-hot type features), bonds = edges; GCN + readout predicts properties |
Connections to Earlier Chapters¶
| Chapter | Connection |
|---|---|
| Ch08 (Clustering) | Community detection is the unsupervised analogue of GCN node classification |
| Ch14 (Neural Networks) | Same AdamW training loop, cross-entropy loss, batch gradient descent |
| Ch16 (Transformers) | Self-attention is a fully-connected GNN; GCN restricts it to the graph edges |
| Ch17 (BERT) | Pre-train/fine-tune on molecules mirrors BERT pre-train/fine-tune on text |
| Ch18 (ViT) | Patch graph in ViT is a grid graph; GCN generalises to arbitrary topologies |
What's Next?¶
Chapter 20 assembles every tool built across this book into a single working system: a cognitive agent that retrieves medical evidence, propagates confidence through the GCN belief graph you just built, applies an entropy-based refusal threshold, and reflects on its own answer before responding -- built entirely from first principles, no agent framework required.
Section 19.8 — Summary & Quick Reference
Key Formulas¶
| Concept | Formula | Python |
|:--------|:--------|:-------|
| Graph Laplacian | $\mathbf{L} = \mathbf{D} - \mathbf{A}$ | L = D - A |
| Self-loop adjacency | $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$, $\;\tilde{\mathbf{D}}_{ii} = \sum_j \tilde{A}_{ij}$ | A_tilde = A + np.eye(N) |
| Normalised adjacency | $\hat{\mathbf{A}} = \tilde{\mathbf{D}}^{-1/2}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-1/2}$ | D_inv @ A_tilde @ D_inv |
| GCN layer | $\mathbf{H}^{(l+1)} = \sigma\!\left(\hat{\mathbf{A}}\mathbf{H}^{(l)}\mathbf{W}^{(l)}\right)$ | H = relu(A_hat @ H @ W) |
| Graph readout | $\mathbf{z}_{\mathcal{G}} = \tfrac{1}{|\mathcal{V}|}\sum_{v \in \mathcal{V}} \mathbf{h}_v^{(L)}$ | z = H.mean(dim=0) |
ML Connections¶
| Concept | Where It Appears |
|---|---|
| Message passing = attention | Self-attention (Ch16) is a fully-connected GNN; GCN restricts it to graph edges |
| Node classification | Semi-supervised learning with partial labels; analogue of Ch07 classification |
| Graph readout | Pooling over node embeddings; analogue of sequence pooling in Transformers (Ch16) |
| Molecular fingerprints | GCN replaces fixed Morgan fingerprints with differentiable learned atom embeddings |
| Knowledge graph reasoning | RGCN over UMLS / DrugBank; relation-typed edges extend the standard GCN |
What's Next?¶
$$\boxed{\text{Ch 19: Graph Neural Networks}} \;\to\; \boxed{\text{Ch 20: Cognitive Agent}} \;\to\; \boxed{\text{Ch 21: Bridge to Vol II}}$$
Chapter 20 assembles every tool from Chapters 1--19 into a single working system: a cognitive agent that retrieves medical evidence (Ch10 cosine similarity), propagates uncertainty through a GCN belief graph (Ch19), applies an entropy-based refusal threshold (Ch13), and reflects on its own draft before releasing an answer. No agent framework is used -- everything is built from first principles.
References¶
- Kipf TN and Welling M (2017). Semi-supervised classification with graph convolutional networks. ICLR 2017.
- Bruna J, Zaremba W, Szlam A, LeCun Y (2014). Spectral networks and locally connected networks on graphs. ICLR 2014.
- Gilmer J, Schütt KT, Mayr A, et al. (2017). Neural message passing for quantum chemistry (MPNN). ICML, 70, 1263-1272.
- Duvenaud D, Maclaurin D, Aguilera-Iparraguirre J, et al. (2015). Convolutional networks on graphs for learning molecular fingerprints. NeurIPS, 28.
- Hu W, Liu B, Gomes J, et al. (2020). Strategies for pre-training graph neural networks. ICLR 2020.
- Schlichtkrull M, Kipf TN, Bloem P, et al. (2018). Modeling relational data with graph convolutional networks (RGCN). ESWC.
- Jumper J, Evans R, Pritzel A, et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583-589.
- Hamilton WL, Ying R, Leskovec J (2017). Inductive representation learning on large graphs (GraphSAGE). NeurIPS, 30.
- Xu K, Hu W, Leskovec J, Jegelka S (2019). How powerful are graph neural networks? ICLR 2019.
- Velingker A, Golkar S, Bianchini M, et al. (2024). Affinity-based graph networks for molecular property prediction. Journal of Chemical Information and Modeling.