Home / About Us / Volume I Contents / Ch 05: Hypothesis Testing

Statistical Basis of AI/ML — Hypothesis Testing

Chapter 5 — From Sample to Conclusion


Why This Chapter?

In Chapter 4 you learned to quantify uncertainty with probability. Now you will learn to make decisions under uncertainty. Every time a data scientist asks "Is the observed effect real or just noise?" they are performing a hypothesis test — a formal procedure that uses sample data to draw conclusions about an entire population.

Hypothesis testing underpins almost every evaluation in AI/ML:

  • Is the accuracy of Model A significantly better than Model B?
  • Did the new marketing campaign actually increase conversion rates?
  • Is a patient's lab result abnormal relative to the healthy population?

The framework you will build in this chapter follows a universal five-step recipe:

$$\boxed{\text{State } H_0 / H_a} \;\rightarrow\; \boxed{\text{Choose Test}} \;\rightarrow\; \boxed{\text{Set } \alpha} \;\rightarrow\; \boxed{\text{Compute Statistic}} \;\rightarrow\; \boxed{\text{Decide}}$$

You will learn:

  • How the Central Limit Theorem justifies treating sample means as normally distributed.
  • The logic of null and alternate hypotheses, Type I and Type II errors, and $p$-values.
  • When to use a $z$-test versus a $t$-test, one-tailed versus two-tailed tests.
  • Confidence intervals as the decision-making dual of hypothesis tests.
  • Two-sample (independent and paired) $t$-tests.
  • Proportion tests for categorical data.
  • The $\chi^2$ test of variance and the $F$-ratio test for comparing two variances.
  • One-way ANOVA for comparing three or more group means.
  • All of the above implemented in Python with scipy.stats and statsmodels.

What You Will Learn

$\S$ Topic Description
5.0 Setup Imports, configuration, reproducibility
5.1 Sampling Distributions & the CLT Why the mean of means is normal
5.2 Hypothesis Testing Framework $H_0$, $H_a$, $\alpha$, $p$-value, Type I/II errors
5.3 One-Sample $z$-Test Population $\sigma$ known
5.4 Confidence Intervals The dual of the hypothesis test
5.5 One-Sample $t$-Test Population $\sigma$ unknown
5.6 Two-Sample Independent $t$-Test Comparing means from two groups
5.7 Paired $t$-Test Dependent measures (before / after)
5.8 Test of Proportions One-sample and two-sample proportion $z$-tests
5.9 $\chi^2$ Test of Variance Comparing sample variance to a known $\sigma^2$
5.10 $F$-Ratio Test Comparing two sample variances
5.11 One-Way ANOVA Comparing $\geq 3$ group means
5.12 Assumption Checks Shapiro–Wilk (normality) & Levene (equal variance)
5.13 Non-Parametric Alternatives Wilcoxon, Mann–Whitney $U$
5.14 Summary & Quick Reference Decision tree, formulas, Python cheat sheet

Section 5.0 — Setup: Imports & Configuration

This chapter adds statsmodels to the toolkit for proportion tests, ANOVA tables, and Tukey HSD post-hoc comparisons. Run this cell first.

Library Role in This Chapter
NumPy Array arithmetic, random sampling, sqrt
Pandas DataFrames for tabular examples
Matplotlib Distribution curves, rejection-region shading
Seaborn Annotated statistical visualizations
SciPy stats norm, t, chi2, f, ttest_1samp, ttest_ind, f_oneway, shapiro, levene, wilcoxon, mannwhitneyu
statsmodels proportions_ztest, pairwise_tukeyhsd, OLS ANOVA
In [1]:
# === Chapter 5: Setup ===

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import scipy
from scipy import stats
from scipy.stats import (
    norm, t, chi2, f as f_dist,
    ttest_1samp, ttest_ind, ttest_rel,
    f_oneway, shapiro, levene,
    wilcoxon, mannwhitneyu,
)
import statsmodels.api as sm
from statsmodels.stats.proportion import proportions_ztest
from statsmodels.stats.multicomp import pairwise_tukeyhsd

# Plot defaults
plt.rcParams.update({
    "figure.figsize": (8, 4),
    "axes.grid": True,
    "grid.alpha": 0.3,
    "axes.spines.top": False,
    "axes.spines.right": False,
})
sns.set_theme(style="whitegrid")

rng = np.random.default_rng(42)

print("Chapter 5 environment ready.")
print(f"NumPy {np.__version__}  |  Pandas {pd.__version__}  |  SciPy {scipy.__version__}")
Chapter 5 environment ready.
NumPy 2.5.1  |  Pandas 2.3.3  |  SciPy 1.18.0

Section 5.1 — Sampling Distributions & the Central Limit Theorem

Before we can test hypotheses we need to understand why the sample mean behaves so predictably.

Population vs. Sample. A population is the complete set of items of interest (every bottle on the line, every patient in the trial). We almost never observe the whole population, so we draw a sample of size $n$ and compute a statistic — a number summarising the sample (e.g., the sample mean $\bar{x}$). The corresponding true quantity for the entire population is the parameter ($\mu$ for the mean, $\sigma$ for the standard deviation).

Central Limit Theorem (CLT). If we draw many independent samples of size $n$ from any population with mean $\mu$ and finite standard deviation $\sigma$, the distribution of the sample means $\bar{x}$ approaches a normal distribution as $n$ grows:

$$\boxed{\bar{X} \;\sim\; \mathcal{N}\!\left(\mu,\; \frac{\sigma}{\sqrt{n}}\right)}$$

The quantity $\sigma / \sqrt{n}$ is the standard error (SE) of the mean. Key assumptions:

  1. Data are randomly sampled.
  2. Observations are independent (and the sample is $\leq 10\%$ of the population when sampling without replacement).
  3. $n \geq 30$ is generally sufficient for the CLT to hold, even if the population is not normal. The closer the population is to normal, the smaller $n$ can be.
In [2]:
# === Section 5.1: Visual demonstration of the CLT ===
# Roll a fair die (uniform distribution) many times, take means of samples.

pop = np.arange(1, 7)           # faces of a fair die: 1..6
mu_pop = pop.mean()             # 3.5
sigma_pop = pop.std()           # ~1.71 (population SD)

sample_sizes = [5, 30, 100]
n_experiments = 2000

fig, axes = plt.subplots(1, 3, figsize=(14, 4), sharey=True)

for ax, n in zip(axes, sample_sizes):
    means = [rng.choice(pop, size=n, replace=True).mean()
             for _ in range(n_experiments)]
    ax.hist(means, bins=40, density=True, alpha=0.6, color="steelblue",
            edgecolor="white")
    # Overlay the theoretical normal
    se = sigma_pop / np.sqrt(n)
    x = np.linspace(mu_pop - 4*se, mu_pop + 4*se, 200)
    ax.plot(x, norm.pdf(x, mu_pop, se), "r-", lw=2,
            label=f"$N(\\mu={mu_pop},\\,SE={se:.3f})$")
    ax.set_title(f"$n = {n}$", fontsize=13)
    ax.axvline(mu_pop, color="black", ls="--", lw=1)
    ax.legend(fontsize=9)

axes[0].set_ylabel("Density")
fig.suptitle("Central Limit Theorem: Distribution of Sample Means (Fair Die)",
             fontsize=14, y=1.02)
plt.tight_layout()
plt.show()

print(f"Population mean μ = {mu_pop:.2f},  Population SD σ = {sigma_pop:.4f}")
print(f"SE for n=30: σ/√n = {sigma_pop/np.sqrt(30):.4f}")
No description has been provided for this image
Population mean μ = 3.50,  Population SD σ = 1.7078
SE for n=30: σ/√n = 0.3118

Section 5.2 — Hypothesis Testing Framework

A hypothesis is a claim about a population parameter made before looking at the data.

Term Symbol Meaning
Null hypothesis $H_0$ Status quo — "nothing has changed"
Alternate hypothesis $H_a$ (or $H_1$) What the researcher wants to demonstrate
Significance level $\alpha$ Maximum tolerable Type I error rate (commonly 0.05)
$p$-value $p$ Probability of observing data as extreme as ours if $H_0$ is true
Decision rule — Reject $H_0$ if $p < \alpha$; otherwise fail to reject $H_0$

Type I & Type II Errors

$H_0$ is true $H_0$ is false
Reject $H_0$ Type I error ($\alpha$) — false positive Correct (power = $1 - \beta$)
Fail to reject $H_0$ Correct Type II error ($\beta$) — false negative

Analogy: In a courtroom, $H_0$ is "the defendant is innocent." A Type I error convicts an innocent person; a Type II error acquits a guilty one.

Example — Phenylketonuria (PKU) Screening. $H_0$: "Newborn does not have PKU." A false negative (Type II) is far more dangerous than a false positive (Type I), because missing the diagnosis puts the child at lifelong risk — a false positive simply triggers a confirmatory test.

Precision & Recall (ML Bridge). In classification, precision = $\frac{\text{TP}}{\text{TP} + \text{FP}}$ (how many flagged items are truly positive) and recall = $\frac{\text{TP}}{\text{TP} + \text{FN}}$ (how many true positives we caught). The crossover error rate (CER) is the point where Type I and Type II error rates are equal — a lower CER means a more effective system.

The Five-Step Procedure

  1. Develop a clear research question; state $H_0$ and $H_a$.
  2. Choose the appropriate statistical test.
  3. Set $\alpha$ (default 0.05 unless stated otherwise).
  4. Gather data, compute the test statistic and $p$-value.
  5. Decide: if $p < \alpha$ → reject $H_0$; else fail to reject $H_0$.

Which Test?

Data Type Samples Test
Categorical 1 One-proportion $z$-test
Categorical 2 Two-proportion $z$-test
Categorical $\geq 2$ $\chi^2$ test
Continuous, $\sigma$ known 1 $z$-test
Continuous, $\sigma$ unknown 1 One-sample $t$-test
Continuous 2 (independent) Two-sample $t$-test (equal / unequal var.)
Continuous 2 (dependent) Paired $t$-test
Continuous $> 2$ One-way ANOVA

Section 5.3 — One-Sample $z$-Test (Population $\sigma$ Known)

When the population standard deviation $\sigma$ is known and $n$ is large enough for the CLT, the test statistic is:

$$\boxed{z = \frac{\bar{x} - \mu_0}{\sigma \,/\, \sqrt{n}}}$$

One-tailed vs. two-tailed.

| Scenario | $H_a$ | Reject $H_0$ if | |:---------|:------|:----------------| | Two-tailed | $\mu \neq \mu_0$ | $|z| > z_{\alpha/2}$ | | Right-tailed | $\mu > \mu_0$ | $z > z_{\alpha}$ | | Left-tailed | $\mu < \mu_0$ | $z < -z_{\alpha}$ |

Example — Coca-Cola Bottle Volume. A 600 mL bottle has known $\sigma = 5$ mL. A customer claims the mean volume differs from 600 mL. A sample of $n = 36$ gives $\bar{x} = 598.2$ mL. Test at $\alpha = 0.05$.

In [3]:
# === Section 5.3: One-sample z-test — Coca-Cola bottle volume ===

mu_0  = 600      # claimed population mean (mL)
sigma = 5        # known population SD
n     = 36
x_bar = 598.2    # sample mean
alpha = 0.05

# Test statistic
z_stat = (x_bar - mu_0) / (sigma / np.sqrt(n))

# Two-tailed p-value
p_value = 2 * norm.cdf(-abs(z_stat))   # symmetric

z_crit = norm.ppf(1 - alpha / 2)       # ≈ 1.96

print("=== One-Sample z-Test: Coca-Cola Bottle Volume ===")
print(f"H₀: μ = {mu_0} mL    H₁: μ ≠ {mu_0} mL")
print(f"x̄ = {x_bar},  σ = {sigma},  n = {n}")
print(f"z = ({x_bar} − {mu_0}) / ({sigma}/√{n}) = {z_stat:.4f}")
print(f"z-critical (α/2 = {alpha/2}) = ±{z_crit:.4f}")
print(f"p-value = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
print("Conclusion: The sample provides sufficient evidence that the mean "
      "volume differs from 600 mL." if p_value < alpha else
      "Conclusion: Insufficient evidence to claim the mean differs from 600 mL.")
=== One-Sample z-Test: Coca-Cola Bottle Volume ===
H₀: μ = 600 mL    H₁: μ ≠ 600 mL
x̄ = 598.2,  σ = 5,  n = 36
z = (598.2 − 600) / (5/√36) = -2.1600
z-critical (α/2 = 0.025) = ±1.9600
p-value = 0.030773

Decision: Reject H₀
Conclusion: The sample provides sufficient evidence that the mean volume differs from 600 mL.
In [4]:
# === Section 5.3 (cont.): Visualise rejection regions ===

x = np.linspace(-4, 4, 400)
y = norm.pdf(x)

fig, ax = plt.subplots(figsize=(9, 4))
ax.plot(x, y, "k-", lw=2)

# Shade rejection regions
ax.fill_between(x, y, where=(x <= -z_crit), color="red", alpha=0.3,
                label=f"Reject region (α/2 = {alpha/2})")
ax.fill_between(x, y, where=(x >= z_crit), color="red", alpha=0.3)

# Mark test statistic
ax.axvline(z_stat, color="blue", ls="--", lw=2,
           label=f"z-stat = {z_stat:.2f}")
ax.axvline(-z_crit, color="red", ls=":", lw=1)
ax.axvline(z_crit, color="red", ls=":", lw=1)

ax.set_title("Two-Tailed z-Test: Rejection Regions", fontsize=13)
ax.set_xlabel("z")
ax.set_ylabel("Density")
ax.legend()
plt.tight_layout()
plt.show()
No description has been provided for this image

Example — Waiting Time (One-Tailed). A hospital lobby historically has a mean wait of $\mu_0 = 4.5$ minutes with $\sigma = 1.2$ min. A sample of $n = 25$ patients today gives $\bar{x} = 5.1$ min. Has the wait increased? Test at $\alpha = 0.01$.

In [5]:
# === Section 5.3 (cont.): One-tailed z-test — Waiting time ===

mu_0  = 4.5;  sigma = 1.2;  n = 25;  x_bar = 5.1;  alpha = 0.01

z_stat  = (x_bar - mu_0) / (sigma / np.sqrt(n))
p_value = 1 - norm.cdf(z_stat)          # right-tailed
z_crit  = norm.ppf(1 - alpha)            # ≈ 2.326

print("=== One-Tailed z-Test: Hospital Waiting Time ===")
print(f"H₀: μ ≤ {mu_0} min    H₁: μ > {mu_0} min")
print(f"x̄ = {x_bar},  σ = {sigma},  n = {n}")
print(f"z = {z_stat:.4f}    z-critical (α={alpha}) = {z_crit:.4f}")
print(f"p-value = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if z_stat > z_crit else 'Fail to reject H₀'}")
print(f"At α = {alpha}, {x_bar} min is {z_stat:.1f} SDs above {mu_0} — "
      "evidence supports that wait times have increased.")
=== One-Tailed z-Test: Hospital Waiting Time ===
H₀: μ ≤ 4.5 min    H₁: μ > 4.5 min
x̄ = 5.1,  σ = 1.2,  n = 25
z = 2.5000    z-critical (α=0.01) = 2.3263
p-value = 0.006210

Decision: Reject H₀
At α = 0.01, 5.1 min is 2.5 SDs above 4.5 — evidence supports that wait times have increased.

Section 5.4 — Confidence Intervals

A confidence interval answers the same question as a hypothesis test but frames it as a range of plausible values for $\mu$:

$$\boxed{\text{CI}_{1-\alpha} = \bar{x} \;\pm\; z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}}$$

If $\mu_0$ falls inside the CI, we fail to reject $H_0$. If $\mu_0$ falls outside the CI, we reject $H_0$.

Example — Paper Length. A manufacturer claims sheets are 11 inches long. A sample of $n = 100$ gives $\bar{x} = 10.998$ with known $\sigma = 0.02$. Construct a 95 % CI.

In [6]:
# === Section 5.4: Confidence interval — Paper length ===

x_bar = 10.998;  sigma = 0.02;  n = 100;  alpha = 0.05;  mu_0 = 11.0

z_crit = norm.ppf(1 - alpha / 2)          # 1.96
se     = sigma / np.sqrt(n)
ci_lo  = x_bar - z_crit * se
ci_hi  = x_bar + z_crit * se

print("=== 95% Confidence Interval: Paper Length ===")
print(f"x̄ = {x_bar},  σ = {sigma},  n = {n}")
print(f"SE = σ/√n = {se:.5f}")
print(f"95% CI = {x_bar} ± {z_crit:.2f} × {se:.5f}")
print(f"       = [{ci_lo:.5f}, {ci_hi:.5f}]")
print(f"\nμ₀ = {mu_0} is {'inside' if ci_lo <= mu_0 <= ci_hi else 'outside'} the CI.")
print("Conclusion: Cannot reject H₀ that the mean length is 11 inches.")
=== 95% Confidence Interval: Paper Length ===
x̄ = 10.998,  σ = 0.02,  n = 100
SE = σ/√n = 0.00200
95% CI = 10.998 ± 1.96 × 0.00200
       = [10.99408, 11.00192]

μ₀ = 11.0 is inside the CI.
Conclusion: Cannot reject H₀ that the mean length is 11 inches.

Section 5.5 — One-Sample $t$-Test (Population $\sigma$ Unknown)

When $\sigma$ is unknown we estimate it with the sample standard deviation $s$ and use the $t$-distribution (heavier tails than normal, extra uncertainty from estimating $\sigma$):

$$\boxed{t = \frac{\bar{x} - \mu_0}{s \,/\, \sqrt{n}}, \quad df = n - 1}$$

As $n \to \infty$ the $t$-distribution converges to the standard normal.

$\alpha$ $t$-critical ($df = 29$) $t$-critical ($df = 99$) $z$ (normal)
0.05 (two-tailed) 2.045 1.984 1.960

Example — Daily Energy Intake. Eleven women's daily energy intakes (kJ) are recorded. Is the population mean different from the recommended $\mu_0 = 7725$ kJ? ($\alpha = 0.05$, two-tailed.)

In [7]:
# === Section 5.5: One-sample t-test — Energy intake ===

intake = np.array([5260, 5470, 5640, 6180, 6390, 6515,
                    6805, 7515, 7515, 8230, 8770])   # kJ
mu_0  = 7725
alpha = 0.05

t_stat, p_value = ttest_1samp(intake, mu_0)

print("=== One-Sample t-Test: Daily Energy Intake ===")
print(f"n = {len(intake)},  x̄ = {intake.mean():.1f} kJ,  s = {intake.std(ddof=1):.1f} kJ")
print(f"H₀: μ = {mu_0} kJ    H₁: μ ≠ {mu_0} kJ")
print(f"t-statistic = {t_stat:.4f},  df = {len(intake)-1}")
print(f"p-value = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
print("The mean intake is significantly below the recommended 7725 kJ."
      if p_value < alpha else
      "Insufficient evidence that the mean differs from 7725 kJ.")
=== One-Sample t-Test: Daily Energy Intake ===
n = 11,  x̄ = 6753.6 kJ,  s = 1142.1 kJ
H₀: μ = 7725 kJ    H₁: μ ≠ 7725 kJ
t-statistic = -2.8208,  df = 10
p-value = 0.018137

Decision: Reject H₀
The mean intake is significantly below the recommended 7725 kJ.
In [8]:
# === Section 5.5 (cont.): Non-parametric alternative — Wilcoxon signed-rank ===
# Use when normality assumption is doubtful.

z_stat_w, p_value_w = wilcoxon(intake - mu_0)

print("=== Wilcoxon Signed-Rank Test (non-parametric) ===")
print(f"Test statistic = {z_stat_w:.4f},  p-value = {p_value_w:.6f}")
print(f"Decision: {'Reject H₀' if p_value_w < alpha else 'Fail to reject H₀'}")
print("Both the t-test and Wilcoxon agree."
      if (p_value < alpha) == (p_value_w < alpha) else
      "The parametric and non-parametric tests disagree — investigate further.")
=== Wilcoxon Signed-Rank Test (non-parametric) ===
Test statistic = 8.0000,  p-value = 0.022461
Decision: Reject H₀
Both the t-test and Wilcoxon agree.

Section 5.6 — Two-Sample Independent $t$-Test

When two independent groups are sampled and we want to compare their means:

$$\boxed{t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}}}$$

The formula above is Welch's $t$-test (default in scipy), which does not assume equal variances. If variances are equal, a pooled variant is used (equal_var=True).

Degrees of freedom (Welch–Satterthwaite):

$$df = \frac{\left(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}\right)^2}{\frac{(s_1^2/n_1)^2}{n_1-1} + \frac{(s_2^2/n_2)^2}{n_2-1}}$$

Example — Energy Expenditure: Obese vs. Lean. Energy expenditure (MJ) is measured for two independent groups. Is there a difference?

In [9]:
# === Section 5.6: Two-sample independent t-test — Energy expenditure ===

obese = np.array([9.21, 7.53, 7.48, 8.08, 8.09, 10.15, 8.40,
                   10.88, 6.13, 7.90, 11.51, 12.79])
lean  = np.array([7.53, 7.48, 8.08, 8.09, 10.15, 8.40, 10.88,
                   6.13, 7.90, 11.51])

alpha = 0.05

# Step 1: Check equal-variance assumption (Levene's test)
lev_stat, lev_p = levene(obese, lean)
print(f"Levene's test:  F = {lev_stat:.4f},  p = {lev_p:.4f}")
equal_var = lev_p >= alpha
print(f"Equal variances assumed: {equal_var}\n")

# Step 2: Two-sample t-test
t_stat, p_value = ttest_ind(obese, lean, equal_var=equal_var)

print("=== Two-Sample t-Test: Obese vs. Lean ===")
print(f"Obese: n={len(obese)}, x̄={obese.mean():.2f}, s={obese.std(ddof=1):.2f}")
print(f"Lean:  n={len(lean)},  x̄={lean.mean():.2f}, s={lean.std(ddof=1):.2f}")
print(f"t = {t_stat:.4f},  p = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
Levene's test:  F = 0.2444,  p = 0.6264
Equal variances assumed: True

=== Two-Sample t-Test: Obese vs. Lean ===
Obese: n=12, x̄=9.01, s=1.94
Lean:  n=10,  x̄=8.62, s=1.69
t = 0.5067,  p = 0.617923

Decision: Fail to reject H₀
In [10]:
# === Section 5.6 (cont.): Non-parametric alternative — Mann-Whitney U ===

u_stat, u_p = mannwhitneyu(obese, lean, alternative="two-sided")

print("=== Mann-Whitney U Test (non-parametric) ===")
print(f"U = {u_stat:.1f},  p = {u_p:.6f}")
print(f"Decision: {'Reject H₀' if u_p < alpha else 'Fail to reject H₀'}")
=== Mann-Whitney U Test (non-parametric) ===
U = 67.0,  p = 0.667332
Decision: Fail to reject H₀

Section 5.7 — Paired $t$-Test (Dependent Measures)

When the two measurements are not independent (e.g., the same subject measured before and after treatment), we compute the difference $D_i = X_i - Y_i$ for each pair and test whether $\bar{D}$ differs from 0:

$$\boxed{t = \frac{\bar{D}}{s_D / \sqrt{n}}, \quad df = n - 1}$$

Example — Concrete Strength at 2 vs. 7 Days. Concrete strength (MPa) is measured at 2 days and 7 days for 10 samples. Is there a significant difference at $\alpha = 0.01$?

In [11]:
# === Section 5.7: Paired t-test — Concrete strength ===

day2 = np.array([20.5, 18.8, 22.1, 19.6, 21.3, 17.9, 23.0, 20.2, 18.5, 22.8])
day7 = np.array([25.3, 23.1, 27.8, 24.2, 26.5, 22.4, 28.9, 25.0, 23.6, 27.5])
alpha = 0.01

D = day2 - day7

# scipy.stats.ttest_rel computes the paired t-test directly
t_stat, p_value = ttest_rel(day2, day7)

print("=== Paired t-Test: Concrete Strength (2 days vs. 7 days) ===")
print(f"n = {len(D)} paired observations")
print(f"D̄ = {D.mean():.2f} MPa,  s_D = {D.std(ddof=1):.2f} MPa")
print(f"t = {t_stat:.4f},  df = {len(D)-1},  p = {p_value:.8f}")
print(f"\nDecision (α={alpha}): {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
print("Conclusion: Concrete strength at 7 days is significantly greater "
      "than at 2 days." if p_value < alpha else
      "Insufficient evidence of a difference.")
=== Paired t-Test: Concrete Strength (2 days vs. 7 days) ===
n = 10 paired observations
D̄ = -4.96 MPa,  s_D = 0.52 MPa
t = -30.3484,  df = 9,  p = 0.00000000

Decision (α=0.01): Reject H₀
Conclusion: Concrete strength at 7 days is significantly greater than at 2 days.

Section 5.8 — Test of Proportions

For categorical data we test whether a population proportion $p$ equals a hypothesised value $p_0$.

One-Sample Proportion $z$-Test

$$\boxed{z = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}}}$$

Example — Underfilled Bottles. Five years ago 30 % of bottles were underfilled ($p_0 = 0.30$). After corrective measures, 76 of 300 tested bottles are underfilled ($\hat{p} = 0.2533$). Was the correction effective ($\alpha = 0.05$, left-tailed)?

Two-Sample Proportion $z$-Test

$$z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}$$

where $\hat{p} = \frac{x_1 + x_2}{n_1 + n_2}$ is the pooled proportion.

In [12]:
# === Section 5.8: One-sample proportion z-test — Underfilled bottles ===

p0 = 0.30        # historical proportion underfilled
n  = 300
x  = 76           # underfilled in sample
p_hat = x / n
alpha = 0.05

# Manual calculation
z_stat = (p_hat - p0) / np.sqrt(p0 * (1 - p0) / n)
p_value = norm.cdf(z_stat)               # left-tailed

print("=== One-Sample Proportion z-Test: Underfilled Bottles ===")
print(f"H₀: p = {p0}   H₁: p < {p0}")
print(f"p̂ = {x}/{n} = {p_hat:.4f}")
print(f"z = {z_stat:.4f},  p-value = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
print("The corrective measure significantly reduced the underfill rate."
      if p_value < alpha else "Insufficient evidence.")
=== One-Sample Proportion z-Test: Underfilled Bottles ===
H₀: p = 0.3   H₁: p < 0.3
p̂ = 76/300 = 0.2533
z = -1.7638,  p-value = 0.038880

Decision: Reject H₀
The corrective measure significantly reduced the underfill rate.
In [13]:
# === Section 5.8 (cont.): Two-sample proportion z-test ===
# Agency claims 35% of 155 banks have assets > $10 B.
# Sample: 15 of 50 banks have assets > $10 B.
# Test if proportion < 0.35.

count = np.array([15, 155 * 0.35])   # successes in sample vs. claimed
nobs  = np.array([50, 155])

z_stat, p_value = proportions_ztest(count, nobs, alternative="smaller")

print("=== Two-Sample Proportion z-Test: Bank Assets ===")
print(f"Sample: {15}/{50} = {15/50:.2f}   Claimed: {0.35:.2f}")
print(f"z = {z_stat:.4f},  p-value = {p_value:.6f}")
print(f"Decision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
=== Two-Sample Proportion z-Test: Bank Assets ===
Sample: 15/50 = 0.30   Claimed: 0.35
z = -0.6500,  p-value = 0.257844
Decision: Fail to reject H₀

Section 5.9 — $\chi^2$ Test of Variance

The chi-square test compares a sample variance to a known population variance. The $\chi^2$ distribution is not symmetric — its shape depends on the degrees of freedom.

$$\boxed{\chi^2 = \frac{(n-1)\,s^2}{\sigma_0^2}, \quad df = n - 1}$$

If $s = \sigma_0$, then $\chi^2 = n - 1$. A much larger $\chi^2$ indicates the sample variance exceeds the population variance.

Example — Coca-Cola Filling Machine. The specification requires $\sigma = 5$ mL. A sample of 10 bottles gives the following volumes:

In [14]:
# === Section 5.9: Chi-square test of variance — Bottle volumes ===

volumes = np.array([607, 602, 590, 603, 598, 602, 597, 600, 603, 598])
sigma_0 = 5.0     # specified population SD
alpha   = 0.05

n = len(volumes)
s2 = volumes.var(ddof=1)                  # sample variance
chi2_stat = (n - 1) * s2 / sigma_0**2
df = n - 1

# Two-tailed critical values
chi2_lo = chi2.ppf(alpha / 2, df)
chi2_hi = chi2.ppf(1 - alpha / 2, df)

# p-value (two-tailed)
p_lo = chi2.cdf(chi2_stat, df)
p_hi = 1 - chi2.cdf(chi2_stat, df)
p_value = 2 * min(p_lo, p_hi)

print("=== χ² Test of Variance: Bottle Filling ===")
print(f"n = {n},  s² = {s2:.2f},  σ₀² = {sigma_0**2:.1f}")
print(f"χ² = (n−1)·s²/σ₀² = {chi2_stat:.4f},  df = {df}")
print(f"Critical region: χ² < {chi2_lo:.4f} or χ² > {chi2_hi:.4f}")
print(f"p-value ≈ {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if (chi2_stat < chi2_lo or chi2_stat > chi2_hi) else 'Fail to reject H₀'}")
=== χ² Test of Variance: Bottle Filling ===
n = 10,  s² = 21.33,  σ₀² = 25.0
χ² = (n−1)·s²/σ₀² = 7.6800,  df = 9
Critical region: χ² < 2.7004 or χ² > 19.0228
p-value ≈ 0.866624

Decision: Fail to reject H₀

Section 5.10 — $F$-Ratio Test (Comparing Two Variances)

The $F$-test compares the variances of two independent samples. The $F$-distribution is right-tailed only (the ratio $s_1^2 / s_2^2$ is always positive).

$$\boxed{F = \frac{s_1^2}{s_2^2}, \quad df_1 = n_1 - 1,\; df_2 = n_2 - 1}$$

Convention: place the larger variance in the numerator.

$H_0$: $\sigma_1^2 = \sigma_2^2$ — the variances are equal.

Example — Stock Volatility. Stock X ($n_1 = 25$) has sample variance $s_1^2 = 6.52$, stock Y ($n_2 = 22$) has $s_2^2 = 3.47$. Are the volatilities different at $\alpha = 0.05$?

In [15]:
# === Section 5.10: F-ratio test — Stock volatility ===

s1_sq, s2_sq = 6.52, 3.47      # sample variances
n1, n2 = 25, 22
alpha  = 0.05

F_stat = s1_sq / s2_sq          # larger / smaller
df1, df2 = n1 - 1, n2 - 1

# Right-tail p-value (multiply by 2 for two-tailed)
p_value = 2 * (1 - f_dist.cdf(F_stat, df1, df2))
F_crit  = f_dist.ppf(1 - alpha / 2, df1, df2)   # upper critical value

print("=== F-Ratio Test: Stock Volatility ===")
print(f"s₁² = {s1_sq},  n₁ = {n1}   |   s₂² = {s2_sq},  n₂ = {n2}")
print(f"F = {s1_sq}/{s2_sq} = {F_stat:.4f},  df₁ = {df1}, df₂ = {df2}")
print(f"F-critical (α/2={alpha/2}) = {F_crit:.4f}")
print(f"p-value = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if F_stat > F_crit else 'Fail to reject H₀'} "
      f"(variances are {'significantly different' if F_stat > F_crit else 'not significantly different'})")
=== F-Ratio Test: Stock Volatility ===
s₁² = 6.52,  n₁ = 25   |   s₂² = 3.47,  n₂ = 22
F = 6.52/3.47 = 1.8790,  df₁ = 24, df₂ = 21
F-critical (α/2=0.025) = 2.3675
p-value = 0.148488

Decision: Fail to reject H₀ (variances are not significantly different)

Section 5.11 — One-Way ANOVA

When comparing the means of three or more independent groups, the $t$-test is insufficient (multiple comparisons inflate Type I error). Analysis of Variance (ANOVA) tests:

$$H_0: \mu_1 = \mu_2 = \cdots = \mu_k \qquad H_a: \text{at least one mean differs}$$

The $F$-statistic for ANOVA is:

$$\boxed{F = \frac{\text{MS}_{\text{between}}}{\text{MS}_{\text{within}}} = \frac{\text{SS}_{\text{between}} \,/\, (k-1)}{\text{SS}_{\text{within}} \,/\, (N-k)}}$$

Assumptions:

  1. Independent random samples.
  2. Each group is approximately normally distributed.
  3. Equal variances across groups (homoscedasticity).

Important: ANOVA tells you that at least one mean differs, but not which one. A post-hoc test (e.g., Tukey HSD) identifies the specific pairs.

Example — Car Crash Head Pressure. Head-pressure readings (arbitrary units) from crashes involving three car types:

In [16]:
# === Section 5.11: One-way ANOVA — Car crash head pressure ===

compact = np.array([643, 655, 702, 685, 671])
midsize = np.array([469, 427, 525, 512, 497])
fullsz  = np.array([484, 456, 402, 513, 495])
alpha = 0.05

# Step 1: Assumption checks
for name, grp in [("Compact", compact), ("Midsize", midsize), ("Full-size", fullsz)]:
    w, p = shapiro(grp)
    print(f"Shapiro-Wilk ({name}): W={w:.4f}, p={p:.4f}  "
          f"{'✓ Normal' if p >= alpha else '✗ Non-normal'}")

lev_stat, lev_p = levene(compact, midsize, fullsz)
print(f"Levene's test: F={lev_stat:.4f}, p={lev_p:.4f}  "
      f"{'✓ Equal var' if lev_p >= alpha else '✗ Unequal var'}\n")

# Step 2: One-way ANOVA
F_stat, p_value = f_oneway(compact, midsize, fullsz)

print("=== One-Way ANOVA: Car Crash Head Pressure ===")
print(f"Compact:   x̄ = {compact.mean():.1f}")
print(f"Midsize:   x̄ = {midsize.mean():.1f}")
print(f"Full-size: x̄ = {fullsz.mean():.1f}")
print(f"F = {F_stat:.4f},  p = {p_value:.8f}")
print(f"\nDecision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
print("At least one car type has a significantly different mean head pressure."
      if p_value < alpha else "No significant difference detected.")
Shapiro-Wilk (Compact): W=0.9823, p=0.9468  ✓ Normal
Shapiro-Wilk (Midsize): W=0.9366, p=0.6423  ✓ Normal
Shapiro-Wilk (Full-size): W=0.9239, p=0.5556  ✓ Normal
Levene's test: F=0.3565, p=0.7073  ✓ Equal var

=== One-Way ANOVA: Car Crash Head Pressure ===
Compact:   x̄ = 671.2
Midsize:   x̄ = 486.0
Full-size: x̄ = 470.0
F = 47.5673,  p = 0.00000197

Decision: Reject H₀
At least one car type has a significantly different mean head pressure.
In [17]:
# === Section 5.11 (cont.): Post-hoc — Tukey HSD ===

# Combine data for Tukey HSD
data = np.concatenate([compact, midsize, fullsz])
labels = (["Compact"] * len(compact) +
          ["Midsize"] * len(midsize) +
          ["Full-size"] * len(fullsz))

tukey = pairwise_tukeyhsd(data, labels, alpha=alpha)
print("=== Tukey HSD Post-Hoc Comparison ===")
print(tukey)
print("\n'Reject = True' means the pair has significantly different means.")
=== Tukey HSD Post-Hoc Comparison ===
     Multiple Comparison of Means - Tukey HSD, FWER=0.05      
==============================================================
  group1    group2  meandiff p-adj    lower     upper   reject
--------------------------------------------------------------
  Compact Full-size   -201.2    0.0 -262.3765 -140.0235   True
  Compact   Midsize   -185.2    0.0 -246.3765 -124.0235   True
Full-size   Midsize     16.0 0.7693  -45.1765   77.1765  False
--------------------------------------------------------------

'Reject = True' means the pair has significantly different means.
In [18]:
# === Section 5.11 (cont.): Boxplot visualisation ===

df_cars = pd.DataFrame({"Pressure": data, "Car Type": labels})

fig, ax = plt.subplots(figsize=(7, 5))
sns.boxplot(data=df_cars, x="Car Type", y="Pressure", palette="Set2", ax=ax)
sns.stripplot(data=df_cars, x="Car Type", y="Pressure", color=".3",
              size=6, ax=ax)
ax.set_title(f"ANOVA: Car Crash Head Pressure\n"
             f"F = {F_stat:.2f}, p = {p_value:.6f}", fontsize=13)
ax.set_ylabel("Head Pressure (a.u.)")
plt.tight_layout()
plt.show()
/tmp/ipykernel_757/1322324669.py:6: FutureWarning: 

Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.

  sns.boxplot(data=df_cars, x="Car Type", y="Pressure", palette="Set2", ax=ax)
No description has been provided for this image

Section 5.12 — Assumption Checks

Most parametric tests assume:

  1. Normality — the data (or residuals) come from a normal distribution.
  2. Equal variances (homoscedasticity) — required for pooled $t$-test and ANOVA.
Check Test scipy.stats Function Null Hypothesis
Normality Shapiro–Wilk shapiro(x) Data is normally distributed
Equal variance Levene levene(x1, x2, ...) Variances are equal

If normality is violated → use a non-parametric alternative (§5.13). If equal variance is violated → use Welch's $t$-test or Welch's ANOVA.

In [19]:
# === Section 5.12: Assumption checks — worked example ===
# We reuse the energy-intake data from §5.5 to demonstrate.

intake = np.array([5260, 5470, 5640, 6180, 6390, 6515,
                    6805, 7515, 7515, 8230, 8770])

# Shapiro-Wilk normality test
w, p_shap = shapiro(intake)
print(f"Shapiro-Wilk: W = {w:.4f}, p = {p_shap:.4f}")
print(f"  → {'Normal assumption holds' if p_shap >= 0.05 else 'Normality rejected'}\n")

# Visual: Q-Q plot
fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Histogram
axes[0].hist(intake, bins=8, edgecolor="white", color="steelblue", alpha=0.7)
axes[0].set_title("Histogram of Energy Intake")
axes[0].set_xlabel("Intake (kJ)")

# Q-Q plot
from scipy.stats import probplot
probplot(intake, dist="norm", plot=axes[1])
axes[1].set_title("Normal Q-Q Plot")
axes[1].get_lines()[0].set_markerfacecolor("steelblue")
axes[1].get_lines()[0].set_markersize(7)

plt.tight_layout()
plt.show()
Shapiro-Wilk: W = 0.9524, p = 0.6743
  → Normal assumption holds

No description has been provided for this image

Section 5.13 — Non-Parametric Alternatives

When the normality assumption fails, use distribution-free tests that operate on ranks rather than raw values.

Parametric Test Non-Parametric Alternative scipy.stats
One-sample $t$ Wilcoxon signed-rank wilcoxon(x - μ₀)
Two-sample $t$ (independent) Mann–Whitney $U$ mannwhitneyu(x, y)
Paired $t$ Wilcoxon signed-rank wilcoxon(d)
One-way ANOVA Kruskal–Wallis $H$ kruskal(x1, x2, ...)

Non-parametric tests are less powerful (higher $\beta$) when normality actually holds, but they are robust when it does not.

In [20]:
# === Section 5.13: Non-parametric summary — Kruskal-Wallis ===
# Non-parametric alternative to ANOVA using the car crash data from §5.11.

from scipy.stats import kruskal

compact = np.array([643, 655, 702, 685, 671])
midsize = np.array([469, 427, 525, 512, 497])
fullsz  = np.array([484, 456, 402, 513, 495])

h_stat, h_p = kruskal(compact, midsize, fullsz)

print("=== Kruskal-Wallis H Test (non-parametric ANOVA) ===")
print(f"H = {h_stat:.4f},  p = {h_p:.6f}")
print(f"Decision: {'Reject H₀' if h_p < 0.05 else 'Fail to reject H₀'}")
print("This confirms the ANOVA result using a distribution-free method.")
=== Kruskal-Wallis H Test (non-parametric ANOVA) ===
H = 9.6200,  p = 0.008148
Decision: Reject H₀
This confirms the ANOVA result using a distribution-free method.

Section 5.14 — Summary & Quick Reference

Decision Tree: Which Test?

Data type?
├── Categorical ─────── # samples?
│   ├── 1 ──────────── One-proportion z-test
│   ├── 2 ──────────── Two-proportion z-test
│   └── ≥ 2 ─────────── χ² test
│
└── Continuous ────────── σ known?
    ├── Yes ────────────── z-test
    └── No ─────────────── # samples?
        ├── 1 ──────────── One-sample t-test
        ├── 2 (independent) ── Two-sample t-test
        │   └── equal var? ── Pooled / Welch's
        ├── 2 (paired) ──── Paired t-test
        └── > 2 ──────────── One-way ANOVA
            └── post-hoc ── Tukey HSD

Key Formulas

Test Statistic Distribution
$z$-test $z = \dfrac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}$ $N(0,1)$
$t$-test $t = \dfrac{\bar{x} - \mu_0}{s/\sqrt{n}}$ $t(df=n-1)$
Paired $t$ $t = \dfrac{\bar{D}}{s_D/\sqrt{n}}$ $t(df=n-1)$
Proportion $z$ $z = \dfrac{\hat{p} - p_0}{\sqrt{p_0(1-p_0)/n}}$ $N(0,1)$
$\chi^2$ variance $\chi^2 = \dfrac{(n-1)s^2}{\sigma_0^2}$ $\chi^2(df=n-1)$
$F$-ratio $F = s_1^2 / s_2^2$ $F(df_1, df_2)$
ANOVA $F$ $F = MS_{\text{between}} / MS_{\text{within}}$ $F(k-1, N-k)$
Confidence interval $\bar{x} \pm z_{\alpha/2} \cdot \dfrac{\sigma}{\sqrt{n}}$ —

Python Cheat Sheet

from scipy.stats import (norm, t, chi2, f,
    ttest_1samp, ttest_ind, ttest_rel, f_oneway,
    shapiro, levene, wilcoxon, mannwhitneyu, kruskal)
from statsmodels.stats.proportion import proportions_ztest
from statsmodels.stats.multicomp import pairwise_tukeyhsd

# One-sample t:   ttest_1samp(data, popmean)
# Two-sample t:   ttest_ind(a, b, equal_var=False)  # Welch's
# Paired t:       ttest_rel(before, after)
# Proportion z:   proportions_ztest(count, nobs, alternative='smaller')
# ANOVA:          f_oneway(g1, g2, g3)
# Tukey HSD:      pairwise_tukeyhsd(data, groups)
# Normality:      shapiro(x)
# Equal var:      levene(a, b)
# Non-parametric: wilcoxon(x - mu)  |  mannwhitneyu(a, b)  |  kruskal(g1, g2, g3)

Key Takeaways

$$\boxed{\text{If } p < \alpha \text{, reject } H_0. \quad \text{If } p \geq \alpha \text{, fail to reject } H_0.}$$

  • "Fail to reject" is not the same as "accept $H_0$".
  • Always check assumptions (normality, equal variance) before choosing a parametric test.
  • The $p$-value is not the probability that $H_0$ is true — it is the probability of seeing data this extreme if $H_0$ were true.
  • Effect size (e.g., Cohen's $d$) complements the $p$-value by quantifying how large the effect is, not just whether it exists.

Next → Chapter 6: Linear Regression


References

  1. Wasserman, L. (2004). All of Statistics: A Concise Course in Statistical Inference. Springer. Chapters 10-11 cover the hypothesis-testing framework used in this chapter.

  2. Casella, G., & Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury. Rigorous treatment of test statistics, power, and Type I/II error.

  3. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA's statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129-133. The authoritative statement on correct interpretation and misuse of p-values.

  4. Student's t-test; Welch, B. L. (1947). The generalization of Student's problem when several different population variances are involved. Biometrika, 34(1-2), 28-35. Origin of the unequal-variance t-test used by default in modern statistical software.

  5. SciPy Development Team. scipy.stats hypothesis testing reference (ttest_ind, chi2_contingency, mannwhitneyu). https://docs.scipy.org/doc/scipy/reference/stats.html

  6. statsmodels Development Team. statsmodels documentation (power analysis, multiple comparisons). https://www.statsmodels.org/stable/index.html