Home / About Us / Volume I Contents / Ch 13: Probability and Information Theory

Statistical Basis of AI/ML — Probability and Information Theory

Chapter 13 — Entropy, Cross-Entropy, and the Mathematics of Learning


Why This Chapter?

In Chapter 4 you built your probability toolkit: distributions, expectation, the central limit theorem. In Chapter 7 you used a logistic regression classifier that outputs a probability via the sigmoid function — but we treated that output as a black box. In Chapter 9 you watched XGBoost minimise the log-loss at each boosting round, again treating the loss as given. This chapter tears open those black boxes and explains why cross-entropy is the right loss for probabilistic classifiers.

The answer comes from information theory, created by Claude Shannon in 1948. The entropy $H(P)$ of a distribution $P$ measures average uncertainty (or average information content) in bits. If you have a model distribution $Q$ but the true distribution is $P$, the extra bits you waste by encoding $P$-data with a $Q$-code is the KL divergence $KL(P \| Q)$. And the total bits used — your wasted overhead plus the unavoidable minimum — is the cross-entropy $H(P, Q)$. Minimising the cross-entropy loss is therefore precisely minimising the KL divergence between the model and the truth.

Softmax converts an arbitrary score vector into a valid probability distribution over $k$ classes. The numerical stability of this computation — why we subtract the maximum before exponentiating — is not a stylistic choice; it is a guard against overflow that every deep learning framework implements internally.

This chapter culminates in one of the most beautiful results in machine learning: the gradient of the cross-entropy loss with respect to the pre-softmax logits is simply $\hat{\mathbf{p}} - \mathbf{y}$. That one line — difference between prediction and truth — is the signal that drives all of supervised classification.

$$\boxed{H(P)} \;\xrightarrow{\text{model mismatch}}\; \boxed{H(P,Q)} \;=\; H(P) + \boxed{KL(P\|Q)} \;\xrightarrow{\text{minimise}}\; \boxed{\text{softmax gradient}}$$

What You Will Learn

$\S$ Topic Description
13.0 Setup Imports & configuration
13.1 Bayes' Theorem — Disease Screening PPV, base-rate neglect, confusion geometry
13.2 Shannon Entropy — Bernoulli Distribution Entropy curve; maximum entropy
13.3 Cross-Entropy and KL Divergence $H(P,Q) = H(P) + KL(P \| Q)$; coding interpretation
13.4 Softmax and Numerical Stability Overflow guard; log-sum-exp trick
13.5 Gradient of Cross-Entropy Loss Analytic derivation; $\partial L/\partial z_i = \hat{p}_i - y_i$
13.6 Model Calibration Log-loss for calibrated vs overconfident predictions
13.7 Summary & Quick Reference Formula table, ML connections
13.8 References Authoritative sources

Section 13.0 — Setup: Imports & Configuration

This chapter is computation-light but derivation-heavy. The code cells primarily verify analytic results numerically and produce visualisations of information-theoretic quantities.

Library Role in This Chapter
NumPy Array arithmetic for entropy, softmax, gradient calculations
SciPy special softmax reference implementation for cross-checking
scikit-learn metrics log_loss — calibrated vs overconfident comparison
Matplotlib / Seaborn Entropy curve, distribution overlays, calibration plots
In [1]:
# === Section 13.0: Imports & configuration ===
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import seaborn as sns
from scipy.special import softmax as scipy_softmax
from sklearn.metrics import log_loss

rng = np.random.default_rng(42)
sns.set_theme(style="whitegrid", palette="muted")
print("NumPy", np.__version__, "| scikit-learn log_loss ready")
NumPy 2.3.5 | scikit-learn log_loss ready

Section 13.1 — Bayes' Theorem: Disease Screening

Bayes' theorem is the foundational rule for updating belief given evidence:

$$\boxed{P(\text{disease} \mid \text{positive}) = \frac{P(\text{positive} \mid \text{disease})\, P(\text{disease})}{P(\text{positive})}}$$

This quantity — the positive predictive value (PPV) — depends critically on the prior $P(\text{disease})$, the base rate or prevalence. A diagnostic test with 99% sensitivity (true-positive rate) and 99% specificity (true-negative rate) sounds nearly perfect, yet for a disease with 0.1% prevalence the PPV is only ~9%: nine out of ten positive tests are false alarms.

This base-rate neglect failure is not merely an abstract statistical curiosity — it underpins misinterpretation of COVID tests, cancer screening programs, and spam filter outputs. In machine learning, it manifests whenever a model trained on a balanced dataset (50% / 50%) is deployed in a population with 1% class prevalence: the confusion matrix entries change dramatically, and precision collapses even as recall stays high.

The code below sweeps over a range of disease prevalences and plots how the PPV of a fixed (high-quality) test varies — a vivid demonstration of why the prior matters as much as the test quality.

In [2]:
# === Section 13.1: Bayesian PPV across prevalence levels ===
sensitivity = 0.99   # P(positive | disease)
specificity = 0.99   # P(negative | not-disease)

prevalences = np.linspace(0.001, 0.5, 500)

def ppv(prev, sens, spec):
    tp = sens * prev
    fp = (1 - spec) * (1 - prev)
    return tp / (tp + fp)

ppv_vals = ppv(prevalences, sensitivity, specificity)

fig, ax = plt.subplots(figsize=(9, 4))
ax.plot(prevalences * 100, ppv_vals * 100, lw=2, color="steelblue")
ax.axhline(50, ls="--", color="tomato", alpha=0.6, label="PPV = 50%")
ax.set_xlabel("Disease prevalence (%)")
ax.set_ylabel("Positive Predictive Value (%)")
ax.set_title(f"PPV of a 99% sens / 99% spec test vs. prevalence")
ax.legend(); plt.tight_layout(); plt.show()

prev_01 = ppv(0.001, sensitivity, specificity)
print(f"At 0.1% prevalence:   PPV = {prev_01*100:.1f}%")
print(f"At 5.0% prevalence:   PPV = {ppv(0.05, sensitivity, specificity)*100:.1f}%")
No description has been provided for this image
At 0.1% prevalence:   PPV = 9.0%
At 5.0% prevalence:   PPV = 83.9%

Section 13.2 — Shannon Entropy: Bernoulli Distribution

The Shannon entropy of a discrete distribution $P$ measures the average number of bits (or nats, if we use $\ln$) needed to encode a sample from $P$:

$$\boxed{H(P) = -\sum_i p_i \log_2 p_i}$$

(with the convention $0 \log 0 = 0$). For a Bernoulli distribution with parameter $p$ — a coin that shows heads with probability $p$ — this is:

$$H(p) = -p \log_2 p - (1-p) \log_2 (1-p)$$

This function achieves its maximum of 1 bit at $p = 0.5$ (a fair coin — maximum uncertainty) and equals 0 bits at $p = 0$ and $p = 1$ (certain outcome — no information). The maximum entropy principle says that, given only a constraint on the mean, we should model the distribution as having maximum entropy — i.e., we should be no more specific than our data forces us to be. In machine learning, the logistic-regression output naturally maximises entropy subject to the observed empirical mean, which is one way to understand why the sigmoid function arises.

In [3]:
# === Section 13.2: Bernoulli entropy curve ===
def entropy_bernoulli(p):  # Shannon entropy (bits) of Bernoulli(p)
    p = np.clip(p, 1e-12, 1 - 1e-12)
    return -p * np.log2(p) - (1 - p) * np.log2(1 - p)

ps = np.linspace(0, 1, 400)
fig, ax = plt.subplots(figsize=(7, 4))
ax.plot(ps, entropy_bernoulli(ps), lw=2.5, color="steelblue")
ax.axvline(0.5, ls="--", color="tomato", alpha=0.7, label="max entropy at p=0.5")
ax.set_xlabel("p  (heads probability)")
ax.set_ylabel("H(p)  [bits]")
ax.set_title("Shannon entropy of Bernoulli(p)")
ax.legend(); plt.tight_layout(); plt.show()

print(f"H(0.5)   = {entropy_bernoulli(0.5):.4f} bits  (fair coin = maximum)")
print(f"H(0.9)   = {entropy_bernoulli(0.9):.4f} bits")
print(f"H(0.999) = {entropy_bernoulli(0.999):.4f} bits  (near-certain)") 
No description has been provided for this image
H(0.5)   = 1.0000 bits  (fair coin = maximum)
H(0.9)   = 0.4690 bits
H(0.999) = 0.0114 bits  (near-certain)

Section 13.3 — Cross-Entropy and KL Divergence

When we have a true distribution $P$ but use a model $Q$ to encode data, the average bits per symbol is the cross-entropy:

$$\boxed{H(P, Q) = -\sum_i P(i)\log Q(i)}$$

It decomposes cleanly into:

$$\boxed{H(P, Q) = H(P) + KL(P \| Q)}$$

where the Kullback–Leibler divergence (also called relative entropy) is:

$$\boxed{KL(P \| Q) = \sum_i P(i)\log \frac{P(i)}{Q(i)} \;\geq\; 0}$$

$KL(P \| Q) = 0$ if and only if $P = Q$. Because the entropy $H(P)$ is fixed (it depends only on the truth, not the model), minimising the cross-entropy loss $H(P, Q)$ is identical to minimising $KL(P \| Q)$ — i.e., making the model $Q$ as close as possible to the true distribution $P$.

This is the information-theoretic justification for using cross-entropy as the training loss for any classifier that outputs a probability. Maximum likelihood estimation of the parameters of $Q$ is equivalent to cross-entropy minimisation, which is equivalent to KL minimisation. All three views are the same optimisation.

In [4]:
# === Section 13.3: Cross-entropy decomposition H(P,Q) = H(P) + KL(P||Q) ===
def cross_entropy(P, Q, base=np.e):  # H(P,Q) in nats
    P = np.array(P, dtype=float); Q = np.array(Q, dtype=float)
    P = np.clip(P, 1e-15, 1); Q = np.clip(Q, 1e-15, 1)
    return -np.sum(P * np.log(Q))

def entropy(P, base=np.e):  # H(P) in nats
    P = np.clip(np.array(P, dtype=float), 1e-15, 1)
    return -np.sum(P * np.log(P))

def kl_div(P, Q):  # KL(P||Q) in nats
    return cross_entropy(P, Q) - entropy(P)

# Example: 4-class distribution
P_true  = np.array([0.5, 0.3, 0.1, 0.1])
Q_model = np.array([0.4, 0.3, 0.2, 0.1])

H_P   = entropy(P_true)
H_PQ  = cross_entropy(P_true, Q_model)
KL_PQ = kl_div(P_true, Q_model)

print(f"P (true)     = {P_true}")
print(f"Q (model)    = {Q_model}")
print()
print(f"H(P)         = {H_P:.4f} nats")
print(f"H(P,Q)       = {H_PQ:.4f} nats")
print(f"KL(P||Q)     = {KL_PQ:.4f} nats")
print(f"H(P) + KL    = {H_P + KL_PQ:.4f} nats  ← must equal H(P,Q): ✓")

print(f"\nKL(P||Q) ≥ 0: {KL_PQ:.4f}  ✓")
print(f"KL(P||P)     = {kl_div(P_true, P_true):.4e}  ← zero ✓")
P (true)     = [0.5 0.3 0.1 0.1]
Q (model)    = [0.4 0.3 0.2 0.1]

H(P)         = 1.1683 nats
H(P,Q)       = 1.2105 nats
KL(P||Q)     = 0.0423 nats
H(P) + KL    = 1.2105 nats  ← must equal H(P,Q): ✓

KL(P||Q) ≥ 0: 0.0423  ✓
KL(P||P)     = 0.0000e+00  ← zero ✓

Section 13.4 — Softmax and Numerical Stability

The softmax function converts a real-valued score vector $\mathbf{z} \in \mathbb{R}^k$ into a valid probability distribution:

$$\boxed{\sigma(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^k e^{z_j}}}$$

For classification networks, $\mathbf{z}$ is the output of the final linear layer (the logits) and $\sigma(\mathbf{z})$ is the vector of predicted class probabilities. Softmax is a smooth, differentiable approximation to argmax: for very large differences in logit values, it concentrates nearly all probability mass on the largest logit.

The numerical stability problem: If any $z_j$ is large (say, 1000), then $e^{1000}$ overflows to inf in 64-bit floating point. The solution is the log-sum-exp trick: subtract the maximum logit before exponentiating, which leaves the softmax value unchanged (the constant cancels):

$$\boxed{\sigma(\mathbf{z})_i = \frac{e^{z_i - \max(\mathbf{z})}}{\sum_j e^{z_j - \max(\mathbf{z})}}}$$

This is not a numerical trick added for convenience — it is the required implementation for any production classifier. PyTorch's F.softmax, TensorFlow's tf.nn.softmax, and SciPy's special.softmax all implement this internally.

In [5]:
# === Section 13.4: Numerically stable softmax ===
def softmax_naive(z):  # naive - overflows for large z
    e = np.exp(z)
    return e / e.sum()

def softmax_stable(z):  # numerically stable: subtract max first
    z = z - z.max()
    e = np.exp(z)
    return e / e.sum()

z_normal = np.array([2.0, 1.0, 0.1])
z_large  = np.array([1000.0, 999.0, 998.0])

print("Normal logits:")
print(f"  naive:    {softmax_naive(z_normal)}")
print(f"  stable:   {softmax_stable(z_normal)}")
print(f"  scipy:    {scipy_softmax(z_normal)}")

print("\nLarge logits (overflow test):")
with np.errstate(over="ignore"):
    naive_out = softmax_naive(z_large)
print(f"  naive:    {naive_out}   ← nan/inf due to overflow!")
print(f"  stable:   {softmax_stable(z_large)}   ← correct")
print(f"  scipy:    {scipy_softmax(z_large)}")
Normal logits:
  naive:    [0.65900114 0.24243297 0.09856589]
  stable:   [0.65900114 0.24243297 0.09856589]
  scipy:    [0.65900114 0.24243297 0.09856589]

Large logits (overflow test):
  naive:    [nan nan nan]   ← nan/inf due to overflow!
  stable:   [0.66524096 0.24472847 0.09003057]   ← correct
  scipy:    [0.66524096 0.24472847 0.09003057]
C:\Users\HOTMAIL\AppData\Local\Temp\ipykernel_9900\2438326407.py:4: RuntimeWarning: invalid value encountered in divide
  return e / e.sum()
In [6]:
# === Section 13.4 (cont.): Visualise softmax concentration ===
temps = [0.1, 0.5, 1.0, 2.0]   # temperature scaling: σ(z/T)
z_base = np.array([3.0, 2.0, 1.0, 0.5, 0.2])
labels = [f"z{i+1}" for i in range(len(z_base))]

fig, axes = plt.subplots(1, 4, figsize=(13, 3), sharey=True)
for ax, T in zip(axes, temps):
    probs = softmax_stable(z_base / T)
    ax.bar(labels, probs, color="steelblue", edgecolor="k")
    ax.set_title(f"T = {T}")
    ax.set_ylim(0, 1)
    ax.set_ylabel("probability" if T == 0.1 else "")

fig.suptitle("Softmax sharpness vs. temperature  T")
plt.tight_layout(); plt.show()
No description has been provided for this image

Section 13.5 — Gradient of the Cross-Entropy Loss

For a $k$-class problem, let $\mathbf{z}$ be the logits (pre-softmax scores), $\hat{\mathbf{p}} = \sigma(\mathbf{z})$ the softmax probabilities, and $\mathbf{y}$ the one-hot target vector. The cross-entropy loss is:

$$\mathcal{L} = -\sum_i y_i \log \hat{p}_i = H(\mathbf{y}, \hat{\mathbf{p}})$$

The gradient with respect to the logits $\mathbf{z}$ is:

$$\boxed{\frac{\partial \mathcal{L}}{\partial z_i} = \hat{p}_i - y_i}$$

This is the most beautiful result in the chapter. Despite the apparent complexity — softmax, log, sum — the gradient is simply the prediction error: the difference between the model's predicted probability and the true label. For the correct class $j$: $\partial \mathcal{L}/\partial z_j = \hat{p}_j - 1$ (the model must push $\hat{p}_j$ closer to 1). For all other classes: $\partial \mathcal{L}/\partial z_i = \hat{p}_i - 0 = \hat{p}_i$ (push them toward 0). Backpropagation through the final layer of any classification network computes exactly this vector.

In [7]:
# === Section 13.5: Analytic gradient of cross-entropy vs finite difference ===
def ce_loss(z, y_one_hot):  # cross-entropy loss = H(y, softmax(z))
    p   = softmax_stable(z)
    return -np.sum(y_one_hot * np.log(np.clip(p, 1e-15, 1)))

def ce_grad_analytic(z, y_one_hot):  # analytic gradient: dL/dz = p_hat - y
    return softmax_stable(z) - y_one_hot

# 3-class example
z       = np.array([2.5, 1.0, 0.5])
y_true  = np.array([1.0, 0.0, 0.0])   # true class is 0

p_hat   = softmax_stable(z)
grad_a  = ce_grad_analytic(z, y_true)

# Finite-difference gradient check
h       = 1e-5
grad_fd = np.array([(ce_loss(z + h*np.eye(len(z))[i], y_true) -
                     ce_loss(z - h*np.eye(len(z))[i], y_true)) / (2*h)
                    for i in range(len(z))])

print("Softmax probabilities  p̂  :", p_hat.round(4))
print("One-hot target          y  :", y_true)
print("Analytic gradient ∂L/∂z   :", grad_a.round(6))
print("Finite-diff gradient       :", grad_fd.round(6))
print(f"Max |analytic − FD|        : {np.max(np.abs(grad_a - grad_fd)):.2e}  ✓")
Softmax probabilities  p̂  : [0.7361 0.1643 0.0996]
One-hot target          y  : [1. 0. 0.]
Analytic gradient ∂L/∂z   : [-0.263875  0.164252  0.099624]
Finite-diff gradient       : [-0.263875  0.164252  0.099624]
Max |analytic − FD|        : 1.01e-11  ✓

Section 13.6 — Model Calibration and Log-Loss

A calibrated classifier is one where the predicted probability $\hat{p}$ matches the observed frequency of the positive class. If your model says 70% probability on 100 examples, exactly 70 of them should be positive.

The log-loss (cross-entropy loss averaged over a dataset) penalises overconfidence disproportionately:

$$\boxed{\text{log-loss} = -\frac{1}{n}\sum_{i=1}^n \left[y_i \log \hat{p}_i + (1-y_i)\log(1-\hat{p}_i)\right]}$$

A model that predicts 0.999 when it should predict 0.7 incurs a huge penalty on the examples it gets wrong. A calibrated model that hedges appropriately — predicting 0.7 instead of 0.999 — achieves lower log-loss even if its classification accuracy is identical.

This matters in practice: a well-calibrated model is safe to use for expected value calculations (clinical decision support, financial risk), whereas an overconfident model produces unreliable uncertainty estimates. Deep learning models without calibration post-processing are notoriously overconfident; temperature scaling (dividing logits by a scalar $T > 1$) is the standard remedy.

In [8]:
# === Section 13.6: Log-loss — calibrated vs overconfident model ===
n        = 500
y_true   = rng.integers(0, 2, n)           # binary labels

# Calibrated: predictions close to true class probabilities
p_calib  = np.where(y_true == 1,
                    rng.uniform(0.55, 0.80, n),
                    rng.uniform(0.20, 0.45, n))

# Overconfident: pushes toward 0 / 1
p_overc  = np.where(y_true == 1,
                    rng.uniform(0.85, 0.99, n),
                    rng.uniform(0.01, 0.15, n))

ll_calib = log_loss(y_true, p_calib)
ll_overc = log_loss(y_true, p_overc)

print(f"Log-loss — calibrated    : {ll_calib:.4f}")
print(f"Log-loss — overconfident : {ll_overc:.4f}")
print(f"\nThe overconfident model has {'higher' if ll_overc > ll_calib else 'lower'} log-loss")
print("(its wrong predictions are punished much harder)")

# Visualise predicted distributions
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
for ax, p_arr, name, ll in [
        (axes[0], p_calib, "Calibrated",    ll_calib),
        (axes[1], p_overc, "Overconfident", ll_overc)]:
    ax.hist(p_arr[y_true == 1], bins=30, alpha=0.6, label="positive", color="steelblue")
    ax.hist(p_arr[y_true == 0], bins=30, alpha=0.6, label="negative", color="tomato")
    ax.set_title(f"{name}  (log-loss={ll:.3f})")
    ax.set_xlabel("predicted probability"); ax.legend()

plt.suptitle("Predicted probability distributions"); plt.tight_layout(); plt.show()
Log-loss — calibrated    : 0.4047
Log-loss — overconfident : 0.0828

The overconfident model has lower log-loss
(its wrong predictions are punished much harder)
No description has been provided for this image

Section 13.7 — Summary & Quick Reference

Key Formulas

Concept Formula Use in ML
Shannon entropy $H(P) = -\sum p_i\log p_i$ Uncertainty; decision-tree splits
Cross-entropy $H(P,Q) = -\sum P_i\log Q_i$ Standard classification loss
KL divergence $KL(P\|Q) = \sum P_i\log(P_i/Q_i) \geq 0$ Variational inference, distillation
Decomposition $H(P,Q) = H(P) + KL(P\|Q)$ CE minimisation = KL minimisation
Softmax $\hat{p}_i = e^{z_i}/\sum e^{z_j}$ Final layer of classifiers
Log-sum-exp trick $z \leftarrow z - z_{\max}$ before softmax Numerical stability
CE gradient $\partial\mathcal{L}/\partial z_i = \hat{p}_i - y_i$ Backpropagation through softmax
Log-loss $-\frac{1}{n}\sum [y\log\hat{p}+(1-y)\log(1-\hat{p})]$ sklearn log_loss

ML Connections

Concept Where It Appears
Entropy Gini impurity ≈ entropy in decision trees; information gain
Cross-entropy loss Every neural classifier: binary (BCELoss) or multi-class (CrossEntropyLoss)
KL divergence VAE loss; policy-gradient trust regions (PPO KL penalty)
Softmax Final layer of image classifiers, language models (next-token prediction)
CE gradient Last layer backward pass in PyTorch / TensorFlow
Calibration Platt scaling, temperature scaling, Bayesian averaging

What's Next?

$$\boxed{\text{Ch 13: Information Theory}} \;\to\; \boxed{\text{Ch 14: Neural Networks — Architecture}} \;\to\; \boxed{\text{Ch 15: Training Deep Networks}}$$

Chapter 14 assembles all the components built across this book: the matrix algebra of Chapter 10, the chain-rule calculus of Chapter 11, and the cross-entropy loss function derived here, into the forward pass and backward pass of a fully-connected neural network.


References

  1. Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley. The definitive graduate textbook on entropy, KL divergence, and the mathematical foundations of information theory.

  2. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423 and 623–656. The original paper defining entropy and mutual information. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x

  3. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Chapter 4 covers the probabilistic framework for classification; Chapters 1–2 provide a thorough treatment of information-theoretic loss functions.

  4. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 3 (Probability and Information Theory) derives entropy, cross-entropy, and KL divergence in the ML context. https://www.deeplearningbook.org/

  5. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML 2017. Introduces temperature scaling and provides rigorous evaluation of calibration. https://arxiv.org/abs/1706.04599

  6. Wikipedia. Cross entropy. https://en.wikipedia.org/wiki/Cross_entropy

  7. Wikipedia. Kullback–Leibler divergence. https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence

  8. Wikipedia. Softmax function. https://en.wikipedia.org/wiki/Softmax_function

  9. scikit-learn developers. sklearn.metrics.log_loss documentation. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html

  10. SciPy Development Team. scipy.special.softmax documentation. https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.softmax.html