Statistical Basis of AI/ML — Probability and Information Theory
Chapter 13 — Entropy, Cross-Entropy, and the Mathematics of Learning
Why This Chapter?
In Chapter 4 you built your probability toolkit: distributions, expectation, the central limit theorem. In Chapter 7 you used a logistic regression classifier that outputs a probability via the sigmoid function — but we treated that output as a black box. In Chapter 9 you watched XGBoost minimise the log-loss at each boosting round, again treating the loss as given. This chapter tears open those black boxes and explains why cross-entropy is the right loss for probabilistic classifiers.
The answer comes from information theory, created by Claude Shannon in 1948. The entropy $H(P)$ of a distribution $P$ measures average uncertainty (or average information content) in bits. If you have a model distribution $Q$ but the true distribution is $P$, the extra bits you waste by encoding $P$-data with a $Q$-code is the KL divergence $KL(P \| Q)$. And the total bits used — your wasted overhead plus the unavoidable minimum — is the cross-entropy $H(P, Q)$. Minimising the cross-entropy loss is therefore precisely minimising the KL divergence between the model and the truth.
Softmax converts an arbitrary score vector into a valid probability distribution over $k$ classes. The numerical stability of this computation — why we subtract the maximum before exponentiating — is not a stylistic choice; it is a guard against overflow that every deep learning framework implements internally.
This chapter culminates in one of the most beautiful results in machine learning: the gradient of the cross-entropy loss with respect to the pre-softmax logits is simply $\hat{\mathbf{p}} - \mathbf{y}$. That one line — difference between prediction and truth — is the signal that drives all of supervised classification.
$$\boxed{H(P)} \;\xrightarrow{\text{model mismatch}}\; \boxed{H(P,Q)} \;=\; H(P) + \boxed{KL(P\|Q)} \;\xrightarrow{\text{minimise}}\; \boxed{\text{softmax gradient}}$$
What You Will Learn
| $\S$ | Topic | Description |
|---|---|---|
| 13.0 | Setup | Imports & configuration |
| 13.1 | Bayes' Theorem — Disease Screening | PPV, base-rate neglect, confusion geometry |
| 13.2 | Shannon Entropy — Bernoulli Distribution | Entropy curve; maximum entropy |
| 13.3 | Cross-Entropy and KL Divergence | $H(P,Q) = H(P) + KL(P \| Q)$; coding interpretation |
| 13.4 | Softmax and Numerical Stability | Overflow guard; log-sum-exp trick |
| 13.5 | Gradient of Cross-Entropy Loss | Analytic derivation; $\partial L/\partial z_i = \hat{p}_i - y_i$ |
| 13.6 | Model Calibration | Log-loss for calibrated vs overconfident predictions |
| 13.7 | Summary & Quick Reference | Formula table, ML connections |
| 13.8 | References | Authoritative sources |
Section 13.0 — Setup: Imports & Configuration
This chapter is computation-light but derivation-heavy. The code cells primarily verify analytic results numerically and produce visualisations of information-theoretic quantities.
| Library | Role in This Chapter |
|---|---|
| NumPy | Array arithmetic for entropy, softmax, gradient calculations |
SciPy special |
softmax reference implementation for cross-checking |
scikit-learn metrics |
log_loss — calibrated vs overconfident comparison |
| Matplotlib / Seaborn | Entropy curve, distribution overlays, calibration plots |
# === Section 13.0: Imports & configuration ===
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import seaborn as sns
from scipy.special import softmax as scipy_softmax
from sklearn.metrics import log_loss
rng = np.random.default_rng(42)
sns.set_theme(style="whitegrid", palette="muted")
print("NumPy", np.__version__, "| scikit-learn log_loss ready")
Section 13.1 — Bayes' Theorem: Disease Screening
Bayes' theorem is the foundational rule for updating belief given evidence:
$$\boxed{P(\text{disease} \mid \text{positive}) = \frac{P(\text{positive} \mid \text{disease})\, P(\text{disease})}{P(\text{positive})}}$$
This quantity — the positive predictive value (PPV) — depends critically on the prior $P(\text{disease})$, the base rate or prevalence. A diagnostic test with 99% sensitivity (true-positive rate) and 99% specificity (true-negative rate) sounds nearly perfect, yet for a disease with 0.1% prevalence the PPV is only ~9%: nine out of ten positive tests are false alarms.
This base-rate neglect failure is not merely an abstract statistical curiosity — it underpins misinterpretation of COVID tests, cancer screening programs, and spam filter outputs. In machine learning, it manifests whenever a model trained on a balanced dataset (50% / 50%) is deployed in a population with 1% class prevalence: the confusion matrix entries change dramatically, and precision collapses even as recall stays high.
The code below sweeps over a range of disease prevalences and plots how the PPV of a fixed (high-quality) test varies — a vivid demonstration of why the prior matters as much as the test quality.
# === Section 13.1: Bayesian PPV across prevalence levels ===
sensitivity = 0.99 # P(positive | disease)
specificity = 0.99 # P(negative | not-disease)
prevalences = np.linspace(0.001, 0.5, 500)
def ppv(prev, sens, spec):
tp = sens * prev
fp = (1 - spec) * (1 - prev)
return tp / (tp + fp)
ppv_vals = ppv(prevalences, sensitivity, specificity)
fig, ax = plt.subplots(figsize=(9, 4))
ax.plot(prevalences * 100, ppv_vals * 100, lw=2, color="steelblue")
ax.axhline(50, ls="--", color="tomato", alpha=0.6, label="PPV = 50%")
ax.set_xlabel("Disease prevalence (%)")
ax.set_ylabel("Positive Predictive Value (%)")
ax.set_title(f"PPV of a 99% sens / 99% spec test vs. prevalence")
ax.legend(); plt.tight_layout(); plt.show()
prev_01 = ppv(0.001, sensitivity, specificity)
print(f"At 0.1% prevalence: PPV = {prev_01*100:.1f}%")
print(f"At 5.0% prevalence: PPV = {ppv(0.05, sensitivity, specificity)*100:.1f}%")
Section 13.2 — Shannon Entropy: Bernoulli Distribution
The Shannon entropy of a discrete distribution $P$ measures the average number of bits (or nats, if we use $\ln$) needed to encode a sample from $P$:
$$\boxed{H(P) = -\sum_i p_i \log_2 p_i}$$
(with the convention $0 \log 0 = 0$). For a Bernoulli distribution with parameter $p$ — a coin that shows heads with probability $p$ — this is:
$$H(p) = -p \log_2 p - (1-p) \log_2 (1-p)$$
This function achieves its maximum of 1 bit at $p = 0.5$ (a fair coin — maximum uncertainty) and equals 0 bits at $p = 0$ and $p = 1$ (certain outcome — no information). The maximum entropy principle says that, given only a constraint on the mean, we should model the distribution as having maximum entropy — i.e., we should be no more specific than our data forces us to be. In machine learning, the logistic-regression output naturally maximises entropy subject to the observed empirical mean, which is one way to understand why the sigmoid function arises.
# === Section 13.2: Bernoulli entropy curve ===
def entropy_bernoulli(p): # Shannon entropy (bits) of Bernoulli(p)
p = np.clip(p, 1e-12, 1 - 1e-12)
return -p * np.log2(p) - (1 - p) * np.log2(1 - p)
ps = np.linspace(0, 1, 400)
fig, ax = plt.subplots(figsize=(7, 4))
ax.plot(ps, entropy_bernoulli(ps), lw=2.5, color="steelblue")
ax.axvline(0.5, ls="--", color="tomato", alpha=0.7, label="max entropy at p=0.5")
ax.set_xlabel("p (heads probability)")
ax.set_ylabel("H(p) [bits]")
ax.set_title("Shannon entropy of Bernoulli(p)")
ax.legend(); plt.tight_layout(); plt.show()
print(f"H(0.5) = {entropy_bernoulli(0.5):.4f} bits (fair coin = maximum)")
print(f"H(0.9) = {entropy_bernoulli(0.9):.4f} bits")
print(f"H(0.999) = {entropy_bernoulli(0.999):.4f} bits (near-certain)")
Section 13.3 — Cross-Entropy and KL Divergence
When we have a true distribution $P$ but use a model $Q$ to encode data, the average bits per symbol is the cross-entropy:
$$\boxed{H(P, Q) = -\sum_i P(i)\log Q(i)}$$
It decomposes cleanly into:
$$\boxed{H(P, Q) = H(P) + KL(P \| Q)}$$
where the Kullback–Leibler divergence (also called relative entropy) is:
$$\boxed{KL(P \| Q) = \sum_i P(i)\log \frac{P(i)}{Q(i)} \;\geq\; 0}$$
$KL(P \| Q) = 0$ if and only if $P = Q$. Because the entropy $H(P)$ is fixed (it depends only on the truth, not the model), minimising the cross-entropy loss $H(P, Q)$ is identical to minimising $KL(P \| Q)$ — i.e., making the model $Q$ as close as possible to the true distribution $P$.
This is the information-theoretic justification for using cross-entropy as the training loss for any classifier that outputs a probability. Maximum likelihood estimation of the parameters of $Q$ is equivalent to cross-entropy minimisation, which is equivalent to KL minimisation. All three views are the same optimisation.
# === Section 13.3: Cross-entropy decomposition H(P,Q) = H(P) + KL(P||Q) ===
def cross_entropy(P, Q, base=np.e): # H(P,Q) in nats
P = np.array(P, dtype=float); Q = np.array(Q, dtype=float)
P = np.clip(P, 1e-15, 1); Q = np.clip(Q, 1e-15, 1)
return -np.sum(P * np.log(Q))
def entropy(P, base=np.e): # H(P) in nats
P = np.clip(np.array(P, dtype=float), 1e-15, 1)
return -np.sum(P * np.log(P))
def kl_div(P, Q): # KL(P||Q) in nats
return cross_entropy(P, Q) - entropy(P)
# Example: 4-class distribution
P_true = np.array([0.5, 0.3, 0.1, 0.1])
Q_model = np.array([0.4, 0.3, 0.2, 0.1])
H_P = entropy(P_true)
H_PQ = cross_entropy(P_true, Q_model)
KL_PQ = kl_div(P_true, Q_model)
print(f"P (true) = {P_true}")
print(f"Q (model) = {Q_model}")
print()
print(f"H(P) = {H_P:.4f} nats")
print(f"H(P,Q) = {H_PQ:.4f} nats")
print(f"KL(P||Q) = {KL_PQ:.4f} nats")
print(f"H(P) + KL = {H_P + KL_PQ:.4f} nats ← must equal H(P,Q): ✓")
print(f"\nKL(P||Q) ≥ 0: {KL_PQ:.4f} ✓")
print(f"KL(P||P) = {kl_div(P_true, P_true):.4e} ← zero ✓")
Section 13.4 — Softmax and Numerical Stability
The softmax function converts a real-valued score vector $\mathbf{z} \in \mathbb{R}^k$ into a valid probability distribution:
$$\boxed{\sigma(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^k e^{z_j}}}$$
For classification networks, $\mathbf{z}$ is the output of the final linear layer (the logits) and $\sigma(\mathbf{z})$ is the vector of predicted class probabilities. Softmax is a smooth, differentiable approximation to argmax: for very large differences in logit values, it concentrates nearly all probability mass on the largest logit.
The numerical stability problem: If any $z_j$ is large (say, 1000), then
$e^{1000}$ overflows to inf in 64-bit floating point. The solution is the
log-sum-exp trick: subtract the maximum logit before exponentiating, which
leaves the softmax value unchanged (the constant cancels):
$$\boxed{\sigma(\mathbf{z})_i = \frac{e^{z_i - \max(\mathbf{z})}}{\sum_j e^{z_j - \max(\mathbf{z})}}}$$
This is not a numerical trick added for convenience — it is the required
implementation for any production classifier. PyTorch's F.softmax, TensorFlow's
tf.nn.softmax, and SciPy's special.softmax all implement this internally.
# === Section 13.4: Numerically stable softmax ===
def softmax_naive(z): # naive - overflows for large z
e = np.exp(z)
return e / e.sum()
def softmax_stable(z): # numerically stable: subtract max first
z = z - z.max()
e = np.exp(z)
return e / e.sum()
z_normal = np.array([2.0, 1.0, 0.1])
z_large = np.array([1000.0, 999.0, 998.0])
print("Normal logits:")
print(f" naive: {softmax_naive(z_normal)}")
print(f" stable: {softmax_stable(z_normal)}")
print(f" scipy: {scipy_softmax(z_normal)}")
print("\nLarge logits (overflow test):")
with np.errstate(over="ignore"):
naive_out = softmax_naive(z_large)
print(f" naive: {naive_out} ← nan/inf due to overflow!")
print(f" stable: {softmax_stable(z_large)} ← correct")
print(f" scipy: {scipy_softmax(z_large)}")
# === Section 13.4 (cont.): Visualise softmax concentration ===
temps = [0.1, 0.5, 1.0, 2.0] # temperature scaling: σ(z/T)
z_base = np.array([3.0, 2.0, 1.0, 0.5, 0.2])
labels = [f"z{i+1}" for i in range(len(z_base))]
fig, axes = plt.subplots(1, 4, figsize=(13, 3), sharey=True)
for ax, T in zip(axes, temps):
probs = softmax_stable(z_base / T)
ax.bar(labels, probs, color="steelblue", edgecolor="k")
ax.set_title(f"T = {T}")
ax.set_ylim(0, 1)
ax.set_ylabel("probability" if T == 0.1 else "")
fig.suptitle("Softmax sharpness vs. temperature T")
plt.tight_layout(); plt.show()
Section 13.5 — Gradient of the Cross-Entropy Loss
For a $k$-class problem, let $\mathbf{z}$ be the logits (pre-softmax scores), $\hat{\mathbf{p}} = \sigma(\mathbf{z})$ the softmax probabilities, and $\mathbf{y}$ the one-hot target vector. The cross-entropy loss is:
$$\mathcal{L} = -\sum_i y_i \log \hat{p}_i = H(\mathbf{y}, \hat{\mathbf{p}})$$
The gradient with respect to the logits $\mathbf{z}$ is:
$$\boxed{\frac{\partial \mathcal{L}}{\partial z_i} = \hat{p}_i - y_i}$$
This is the most beautiful result in the chapter. Despite the apparent complexity — softmax, log, sum — the gradient is simply the prediction error: the difference between the model's predicted probability and the true label. For the correct class $j$: $\partial \mathcal{L}/\partial z_j = \hat{p}_j - 1$ (the model must push $\hat{p}_j$ closer to 1). For all other classes: $\partial \mathcal{L}/\partial z_i = \hat{p}_i - 0 = \hat{p}_i$ (push them toward 0). Backpropagation through the final layer of any classification network computes exactly this vector.
# === Section 13.5: Analytic gradient of cross-entropy vs finite difference ===
def ce_loss(z, y_one_hot): # cross-entropy loss = H(y, softmax(z))
p = softmax_stable(z)
return -np.sum(y_one_hot * np.log(np.clip(p, 1e-15, 1)))
def ce_grad_analytic(z, y_one_hot): # analytic gradient: dL/dz = p_hat - y
return softmax_stable(z) - y_one_hot
# 3-class example
z = np.array([2.5, 1.0, 0.5])
y_true = np.array([1.0, 0.0, 0.0]) # true class is 0
p_hat = softmax_stable(z)
grad_a = ce_grad_analytic(z, y_true)
# Finite-difference gradient check
h = 1e-5
grad_fd = np.array([(ce_loss(z + h*np.eye(len(z))[i], y_true) -
ce_loss(z - h*np.eye(len(z))[i], y_true)) / (2*h)
for i in range(len(z))])
print("Softmax probabilities p̂ :", p_hat.round(4))
print("One-hot target y :", y_true)
print("Analytic gradient ∂L/∂z :", grad_a.round(6))
print("Finite-diff gradient :", grad_fd.round(6))
print(f"Max |analytic − FD| : {np.max(np.abs(grad_a - grad_fd)):.2e} ✓")
Section 13.6 — Model Calibration and Log-Loss
A calibrated classifier is one where the predicted probability $\hat{p}$ matches the observed frequency of the positive class. If your model says 70% probability on 100 examples, exactly 70 of them should be positive.
The log-loss (cross-entropy loss averaged over a dataset) penalises overconfidence disproportionately:
$$\boxed{\text{log-loss} = -\frac{1}{n}\sum_{i=1}^n \left[y_i \log \hat{p}_i + (1-y_i)\log(1-\hat{p}_i)\right]}$$
A model that predicts 0.999 when it should predict 0.7 incurs a huge penalty on the examples it gets wrong. A calibrated model that hedges appropriately — predicting 0.7 instead of 0.999 — achieves lower log-loss even if its classification accuracy is identical.
This matters in practice: a well-calibrated model is safe to use for expected value calculations (clinical decision support, financial risk), whereas an overconfident model produces unreliable uncertainty estimates. Deep learning models without calibration post-processing are notoriously overconfident; temperature scaling (dividing logits by a scalar $T > 1$) is the standard remedy.
# === Section 13.6: Log-loss — calibrated vs overconfident model ===
n = 500
y_true = rng.integers(0, 2, n) # binary labels
# Calibrated: predictions close to true class probabilities
p_calib = np.where(y_true == 1,
rng.uniform(0.55, 0.80, n),
rng.uniform(0.20, 0.45, n))
# Overconfident: pushes toward 0 / 1
p_overc = np.where(y_true == 1,
rng.uniform(0.85, 0.99, n),
rng.uniform(0.01, 0.15, n))
ll_calib = log_loss(y_true, p_calib)
ll_overc = log_loss(y_true, p_overc)
print(f"Log-loss — calibrated : {ll_calib:.4f}")
print(f"Log-loss — overconfident : {ll_overc:.4f}")
print(f"\nThe overconfident model has {'higher' if ll_overc > ll_calib else 'lower'} log-loss")
print("(its wrong predictions are punished much harder)")
# Visualise predicted distributions
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
for ax, p_arr, name, ll in [
(axes[0], p_calib, "Calibrated", ll_calib),
(axes[1], p_overc, "Overconfident", ll_overc)]:
ax.hist(p_arr[y_true == 1], bins=30, alpha=0.6, label="positive", color="steelblue")
ax.hist(p_arr[y_true == 0], bins=30, alpha=0.6, label="negative", color="tomato")
ax.set_title(f"{name} (log-loss={ll:.3f})")
ax.set_xlabel("predicted probability"); ax.legend()
plt.suptitle("Predicted probability distributions"); plt.tight_layout(); plt.show()
Section 13.7 — Summary & Quick Reference
Key Formulas¶
| Concept | Formula | Use in ML |
|---|---|---|
| Shannon entropy | $H(P) = -\sum p_i\log p_i$ | Uncertainty; decision-tree splits |
| Cross-entropy | $H(P,Q) = -\sum P_i\log Q_i$ | Standard classification loss |
| KL divergence | $KL(P\|Q) = \sum P_i\log(P_i/Q_i) \geq 0$ | Variational inference, distillation |
| Decomposition | $H(P,Q) = H(P) + KL(P\|Q)$ | CE minimisation = KL minimisation |
| Softmax | $\hat{p}_i = e^{z_i}/\sum e^{z_j}$ | Final layer of classifiers |
| Log-sum-exp trick | $z \leftarrow z - z_{\max}$ before softmax | Numerical stability |
| CE gradient | $\partial\mathcal{L}/\partial z_i = \hat{p}_i - y_i$ | Backpropagation through softmax |
| Log-loss | $-\frac{1}{n}\sum [y\log\hat{p}+(1-y)\log(1-\hat{p})]$ | sklearn log_loss |
ML Connections¶
| Concept | Where It Appears |
|---|---|
| Entropy | Gini impurity ≈ entropy in decision trees; information gain |
| Cross-entropy loss | Every neural classifier: binary (BCELoss) or multi-class (CrossEntropyLoss) |
| KL divergence | VAE loss; policy-gradient trust regions (PPO KL penalty) |
| Softmax | Final layer of image classifiers, language models (next-token prediction) |
| CE gradient | Last layer backward pass in PyTorch / TensorFlow |
| Calibration | Platt scaling, temperature scaling, Bayesian averaging |
What's Next?¶
$$\boxed{\text{Ch 13: Information Theory}} \;\to\; \boxed{\text{Ch 14: Neural Networks — Architecture}} \;\to\; \boxed{\text{Ch 15: Training Deep Networks}}$$
Chapter 14 assembles all the components built across this book: the matrix algebra of Chapter 10, the chain-rule calculus of Chapter 11, and the cross-entropy loss function derived here, into the forward pass and backward pass of a fully-connected neural network.
References¶
Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley. The definitive graduate textbook on entropy, KL divergence, and the mathematical foundations of information theory.
Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423 and 623–656. The original paper defining entropy and mutual information. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Chapter 4 covers the probabilistic framework for classification; Chapters 1–2 provide a thorough treatment of information-theoretic loss functions.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 3 (Probability and Information Theory) derives entropy, cross-entropy, and KL divergence in the ML context. https://www.deeplearningbook.org/
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML 2017. Introduces temperature scaling and provides rigorous evaluation of calibration. https://arxiv.org/abs/1706.04599
Wikipedia. Cross entropy. https://en.wikipedia.org/wiki/Cross_entropy
Wikipedia. Kullback–Leibler divergence. https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
Wikipedia. Softmax function. https://en.wikipedia.org/wiki/Softmax_function
scikit-learn developers. sklearn.metrics.log_loss documentation. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html
SciPy Development Team. scipy.special.softmax documentation. https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.softmax.html