Statistical Basis of AI/ML — Hypothesis Testing
Chapter 5 — From Sample to Conclusion
Why This Chapter?
In Chapter 4 you learned to quantify uncertainty with probability. Now you will learn to make decisions under uncertainty. Every time a data scientist asks "Is the observed effect real or just noise?" they are performing a hypothesis test — a formal procedure that uses sample data to draw conclusions about an entire population.
Hypothesis testing underpins almost every evaluation in AI/ML:
- Is the accuracy of Model A significantly better than Model B?
- Did the new marketing campaign actually increase conversion rates?
- Is a patient's lab result abnormal relative to the healthy population?
The framework you will build in this chapter follows a universal five-step recipe:
$$\boxed{\text{State } H_0 / H_a} \;\rightarrow\; \boxed{\text{Choose Test}} \;\rightarrow\; \boxed{\text{Set } \alpha} \;\rightarrow\; \boxed{\text{Compute Statistic}} \;\rightarrow\; \boxed{\text{Decide}}$$
You will learn:
- How the Central Limit Theorem justifies treating sample means as normally distributed.
- The logic of null and alternate hypotheses, Type I and Type II errors, and $p$-values.
- When to use a $z$-test versus a $t$-test, one-tailed versus two-tailed tests.
- Confidence intervals as the decision-making dual of hypothesis tests.
- Two-sample (independent and paired) $t$-tests.
- Proportion tests for categorical data.
- The $\chi^2$ test of variance and the $F$-ratio test for comparing two variances.
- One-way ANOVA for comparing three or more group means.
- All of the above implemented in Python with
scipy.statsandstatsmodels.
What You Will Learn
| $\S$ | Topic | Description |
|---|---|---|
| 5.0 | Setup | Imports, configuration, reproducibility |
| 5.1 | Sampling Distributions & the CLT | Why the mean of means is normal |
| 5.2 | Hypothesis Testing Framework | $H_0$, $H_a$, $\alpha$, $p$-value, Type I/II errors |
| 5.3 | One-Sample $z$-Test | Population $\sigma$ known |
| 5.4 | Confidence Intervals | The dual of the hypothesis test |
| 5.5 | One-Sample $t$-Test | Population $\sigma$ unknown |
| 5.6 | Two-Sample Independent $t$-Test | Comparing means from two groups |
| 5.7 | Paired $t$-Test | Dependent measures (before / after) |
| 5.8 | Test of Proportions | One-sample and two-sample proportion $z$-tests |
| 5.9 | $\chi^2$ Test of Variance | Comparing sample variance to a known $\sigma^2$ |
| 5.10 | $F$-Ratio Test | Comparing two sample variances |
| 5.11 | One-Way ANOVA | Comparing $\geq 3$ group means |
| 5.12 | Assumption Checks | Shapiro–Wilk (normality) & Levene (equal variance) |
| 5.13 | Non-Parametric Alternatives | Wilcoxon, Mann–Whitney $U$ |
| 5.14 | Summary & Quick Reference | Decision tree, formulas, Python cheat sheet |
Section 5.0 — Setup: Imports & Configuration
This chapter adds statsmodels to the toolkit for proportion tests,
ANOVA tables, and Tukey HSD post-hoc comparisons. Run this cell first.
| Library | Role in This Chapter |
|---|---|
| NumPy | Array arithmetic, random sampling, sqrt |
| Pandas | DataFrames for tabular examples |
| Matplotlib | Distribution curves, rejection-region shading |
| Seaborn | Annotated statistical visualizations |
SciPy stats |
norm, t, chi2, f, ttest_1samp, ttest_ind, f_oneway, shapiro, levene, wilcoxon, mannwhitneyu |
statsmodels |
proportions_ztest, pairwise_tukeyhsd, OLS ANOVA |
# === Chapter 5: Setup ===
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import scipy
from scipy import stats
from scipy.stats import (
norm, t, chi2, f as f_dist,
ttest_1samp, ttest_ind, ttest_rel,
f_oneway, shapiro, levene,
wilcoxon, mannwhitneyu,
)
import statsmodels.api as sm
from statsmodels.stats.proportion import proportions_ztest
from statsmodels.stats.multicomp import pairwise_tukeyhsd
# Plot defaults
plt.rcParams.update({
"figure.figsize": (8, 4),
"axes.grid": True,
"grid.alpha": 0.3,
"axes.spines.top": False,
"axes.spines.right": False,
})
sns.set_theme(style="whitegrid")
rng = np.random.default_rng(42)
print("Chapter 5 environment ready.")
print(f"NumPy {np.__version__} | Pandas {pd.__version__} | SciPy {scipy.__version__}")
Section 5.1 — Sampling Distributions & the Central Limit Theorem
Before we can test hypotheses we need to understand why the sample mean behaves so predictably.
Population vs. Sample. A population is the complete set of items of interest (every bottle on the line, every patient in the trial). We almost never observe the whole population, so we draw a sample of size $n$ and compute a statistic — a number summarising the sample (e.g., the sample mean $\bar{x}$). The corresponding true quantity for the entire population is the parameter ($\mu$ for the mean, $\sigma$ for the standard deviation).
Central Limit Theorem (CLT). If we draw many independent samples of size $n$ from any population with mean $\mu$ and finite standard deviation $\sigma$, the distribution of the sample means $\bar{x}$ approaches a normal distribution as $n$ grows:
$$\boxed{\bar{X} \;\sim\; \mathcal{N}\!\left(\mu,\; \frac{\sigma}{\sqrt{n}}\right)}$$
The quantity $\sigma / \sqrt{n}$ is the standard error (SE) of the mean. Key assumptions:
- Data are randomly sampled.
- Observations are independent (and the sample is $\leq 10\%$ of the population when sampling without replacement).
- $n \geq 30$ is generally sufficient for the CLT to hold, even if the population is not normal. The closer the population is to normal, the smaller $n$ can be.
# === Section 5.1: Visual demonstration of the CLT ===
# Roll a fair die (uniform distribution) many times, take means of samples.
pop = np.arange(1, 7) # faces of a fair die: 1..6
mu_pop = pop.mean() # 3.5
sigma_pop = pop.std() # ~1.71 (population SD)
sample_sizes = [5, 30, 100]
n_experiments = 2000
fig, axes = plt.subplots(1, 3, figsize=(14, 4), sharey=True)
for ax, n in zip(axes, sample_sizes):
means = [rng.choice(pop, size=n, replace=True).mean()
for _ in range(n_experiments)]
ax.hist(means, bins=40, density=True, alpha=0.6, color="steelblue",
edgecolor="white")
# Overlay the theoretical normal
se = sigma_pop / np.sqrt(n)
x = np.linspace(mu_pop - 4*se, mu_pop + 4*se, 200)
ax.plot(x, norm.pdf(x, mu_pop, se), "r-", lw=2,
label=f"$N(\\mu={mu_pop},\\,SE={se:.3f})$")
ax.set_title(f"$n = {n}$", fontsize=13)
ax.axvline(mu_pop, color="black", ls="--", lw=1)
ax.legend(fontsize=9)
axes[0].set_ylabel("Density")
fig.suptitle("Central Limit Theorem: Distribution of Sample Means (Fair Die)",
fontsize=14, y=1.02)
plt.tight_layout()
plt.show()
print(f"Population mean μ = {mu_pop:.2f}, Population SD σ = {sigma_pop:.4f}")
print(f"SE for n=30: σ/√n = {sigma_pop/np.sqrt(30):.4f}")
Section 5.2 — Hypothesis Testing Framework
A hypothesis is a claim about a population parameter made before looking at the data.
| Term | Symbol | Meaning |
|---|---|---|
| Null hypothesis | $H_0$ | Status quo — "nothing has changed" |
| Alternate hypothesis | $H_a$ (or $H_1$) | What the researcher wants to demonstrate |
| Significance level | $\alpha$ | Maximum tolerable Type I error rate (commonly 0.05) |
| $p$-value | $p$ | Probability of observing data as extreme as ours if $H_0$ is true |
| Decision rule | — | Reject $H_0$ if $p < \alpha$; otherwise fail to reject $H_0$ |
Type I & Type II Errors
| $H_0$ is true | $H_0$ is false | |
|---|---|---|
| Reject $H_0$ | Type I error ($\alpha$) — false positive | Correct (power = $1 - \beta$) |
| Fail to reject $H_0$ | Correct | Type II error ($\beta$) — false negative |
Analogy: In a courtroom, $H_0$ is "the defendant is innocent." A Type I error convicts an innocent person; a Type II error acquits a guilty one.
Example — Phenylketonuria (PKU) Screening. $H_0$: "Newborn does not have PKU." A false negative (Type II) is far more dangerous than a false positive (Type I), because missing the diagnosis puts the child at lifelong risk — a false positive simply triggers a confirmatory test.
Precision & Recall (ML Bridge). In classification, precision = $\frac{\text{TP}}{\text{TP} + \text{FP}}$ (how many flagged items are truly positive) and recall = $\frac{\text{TP}}{\text{TP} + \text{FN}}$ (how many true positives we caught). The crossover error rate (CER) is the point where Type I and Type II error rates are equal — a lower CER means a more effective system.
The Five-Step Procedure¶
- Develop a clear research question; state $H_0$ and $H_a$.
- Choose the appropriate statistical test.
- Set $\alpha$ (default 0.05 unless stated otherwise).
- Gather data, compute the test statistic and $p$-value.
- Decide: if $p < \alpha$ → reject $H_0$; else fail to reject $H_0$.
Which Test?¶
| Data Type | Samples | Test |
|---|---|---|
| Categorical | 1 | One-proportion $z$-test |
| Categorical | 2 | Two-proportion $z$-test |
| Categorical | $\geq 2$ | $\chi^2$ test |
| Continuous, $\sigma$ known | 1 | $z$-test |
| Continuous, $\sigma$ unknown | 1 | One-sample $t$-test |
| Continuous | 2 (independent) | Two-sample $t$-test (equal / unequal var.) |
| Continuous | 2 (dependent) | Paired $t$-test |
| Continuous | $> 2$ | One-way ANOVA |
Section 5.3 — One-Sample $z$-Test (Population $\sigma$ Known)
When the population standard deviation $\sigma$ is known and $n$ is large enough for the CLT, the test statistic is:
$$\boxed{z = \frac{\bar{x} - \mu_0}{\sigma \,/\, \sqrt{n}}}$$
One-tailed vs. two-tailed.
| Scenario | $H_a$ | Reject $H_0$ if | |:---------|:------|:----------------| | Two-tailed | $\mu \neq \mu_0$ | $|z| > z_{\alpha/2}$ | | Right-tailed | $\mu > \mu_0$ | $z > z_{\alpha}$ | | Left-tailed | $\mu < \mu_0$ | $z < -z_{\alpha}$ |
Example — Coca-Cola Bottle Volume. A 600 mL bottle has known $\sigma = 5$ mL. A customer claims the mean volume differs from 600 mL. A sample of $n = 36$ gives $\bar{x} = 598.2$ mL. Test at $\alpha = 0.05$.
# === Section 5.3: One-sample z-test — Coca-Cola bottle volume ===
mu_0 = 600 # claimed population mean (mL)
sigma = 5 # known population SD
n = 36
x_bar = 598.2 # sample mean
alpha = 0.05
# Test statistic
z_stat = (x_bar - mu_0) / (sigma / np.sqrt(n))
# Two-tailed p-value
p_value = 2 * norm.cdf(-abs(z_stat)) # symmetric
z_crit = norm.ppf(1 - alpha / 2) # ≈ 1.96
print("=== One-Sample z-Test: Coca-Cola Bottle Volume ===")
print(f"H₀: μ = {mu_0} mL H₁: μ ≠ {mu_0} mL")
print(f"x̄ = {x_bar}, σ = {sigma}, n = {n}")
print(f"z = ({x_bar} − {mu_0}) / ({sigma}/√{n}) = {z_stat:.4f}")
print(f"z-critical (α/2 = {alpha/2}) = ±{z_crit:.4f}")
print(f"p-value = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
print("Conclusion: The sample provides sufficient evidence that the mean "
"volume differs from 600 mL." if p_value < alpha else
"Conclusion: Insufficient evidence to claim the mean differs from 600 mL.")
# === Section 5.3 (cont.): Visualise rejection regions ===
x = np.linspace(-4, 4, 400)
y = norm.pdf(x)
fig, ax = plt.subplots(figsize=(9, 4))
ax.plot(x, y, "k-", lw=2)
# Shade rejection regions
ax.fill_between(x, y, where=(x <= -z_crit), color="red", alpha=0.3,
label=f"Reject region (α/2 = {alpha/2})")
ax.fill_between(x, y, where=(x >= z_crit), color="red", alpha=0.3)
# Mark test statistic
ax.axvline(z_stat, color="blue", ls="--", lw=2,
label=f"z-stat = {z_stat:.2f}")
ax.axvline(-z_crit, color="red", ls=":", lw=1)
ax.axvline(z_crit, color="red", ls=":", lw=1)
ax.set_title("Two-Tailed z-Test: Rejection Regions", fontsize=13)
ax.set_xlabel("z")
ax.set_ylabel("Density")
ax.legend()
plt.tight_layout()
plt.show()
Example — Waiting Time (One-Tailed). A hospital lobby historically has a mean wait of $\mu_0 = 4.5$ minutes with $\sigma = 1.2$ min. A sample of $n = 25$ patients today gives $\bar{x} = 5.1$ min. Has the wait increased? Test at $\alpha = 0.01$.
# === Section 5.3 (cont.): One-tailed z-test — Waiting time ===
mu_0 = 4.5; sigma = 1.2; n = 25; x_bar = 5.1; alpha = 0.01
z_stat = (x_bar - mu_0) / (sigma / np.sqrt(n))
p_value = 1 - norm.cdf(z_stat) # right-tailed
z_crit = norm.ppf(1 - alpha) # ≈ 2.326
print("=== One-Tailed z-Test: Hospital Waiting Time ===")
print(f"H₀: μ ≤ {mu_0} min H₁: μ > {mu_0} min")
print(f"x̄ = {x_bar}, σ = {sigma}, n = {n}")
print(f"z = {z_stat:.4f} z-critical (α={alpha}) = {z_crit:.4f}")
print(f"p-value = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if z_stat > z_crit else 'Fail to reject H₀'}")
print(f"At α = {alpha}, {x_bar} min is {z_stat:.1f} SDs above {mu_0} — "
"evidence supports that wait times have increased.")
Section 5.4 — Confidence Intervals
A confidence interval answers the same question as a hypothesis test but frames it as a range of plausible values for $\mu$:
$$\boxed{\text{CI}_{1-\alpha} = \bar{x} \;\pm\; z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}}$$
If $\mu_0$ falls inside the CI, we fail to reject $H_0$. If $\mu_0$ falls outside the CI, we reject $H_0$.
Example — Paper Length. A manufacturer claims sheets are 11 inches long. A sample of $n = 100$ gives $\bar{x} = 10.998$ with known $\sigma = 0.02$. Construct a 95 % CI.
# === Section 5.4: Confidence interval — Paper length ===
x_bar = 10.998; sigma = 0.02; n = 100; alpha = 0.05; mu_0 = 11.0
z_crit = norm.ppf(1 - alpha / 2) # 1.96
se = sigma / np.sqrt(n)
ci_lo = x_bar - z_crit * se
ci_hi = x_bar + z_crit * se
print("=== 95% Confidence Interval: Paper Length ===")
print(f"x̄ = {x_bar}, σ = {sigma}, n = {n}")
print(f"SE = σ/√n = {se:.5f}")
print(f"95% CI = {x_bar} ± {z_crit:.2f} × {se:.5f}")
print(f" = [{ci_lo:.5f}, {ci_hi:.5f}]")
print(f"\nμ₀ = {mu_0} is {'inside' if ci_lo <= mu_0 <= ci_hi else 'outside'} the CI.")
print("Conclusion: Cannot reject H₀ that the mean length is 11 inches.")
Section 5.5 — One-Sample $t$-Test (Population $\sigma$ Unknown)
When $\sigma$ is unknown we estimate it with the sample standard deviation $s$ and use the $t$-distribution (heavier tails than normal, extra uncertainty from estimating $\sigma$):
$$\boxed{t = \frac{\bar{x} - \mu_0}{s \,/\, \sqrt{n}}, \quad df = n - 1}$$
As $n \to \infty$ the $t$-distribution converges to the standard normal.
| $\alpha$ | $t$-critical ($df = 29$) | $t$-critical ($df = 99$) | $z$ (normal) |
|---|---|---|---|
| 0.05 (two-tailed) | 2.045 | 1.984 | 1.960 |
Example — Daily Energy Intake. Eleven women's daily energy intakes (kJ) are recorded. Is the population mean different from the recommended $\mu_0 = 7725$ kJ? ($\alpha = 0.05$, two-tailed.)
# === Section 5.5: One-sample t-test — Energy intake ===
intake = np.array([5260, 5470, 5640, 6180, 6390, 6515,
6805, 7515, 7515, 8230, 8770]) # kJ
mu_0 = 7725
alpha = 0.05
t_stat, p_value = ttest_1samp(intake, mu_0)
print("=== One-Sample t-Test: Daily Energy Intake ===")
print(f"n = {len(intake)}, x̄ = {intake.mean():.1f} kJ, s = {intake.std(ddof=1):.1f} kJ")
print(f"H₀: μ = {mu_0} kJ H₁: μ ≠ {mu_0} kJ")
print(f"t-statistic = {t_stat:.4f}, df = {len(intake)-1}")
print(f"p-value = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
print("The mean intake is significantly below the recommended 7725 kJ."
if p_value < alpha else
"Insufficient evidence that the mean differs from 7725 kJ.")
# === Section 5.5 (cont.): Non-parametric alternative — Wilcoxon signed-rank ===
# Use when normality assumption is doubtful.
z_stat_w, p_value_w = wilcoxon(intake - mu_0)
print("=== Wilcoxon Signed-Rank Test (non-parametric) ===")
print(f"Test statistic = {z_stat_w:.4f}, p-value = {p_value_w:.6f}")
print(f"Decision: {'Reject H₀' if p_value_w < alpha else 'Fail to reject H₀'}")
print("Both the t-test and Wilcoxon agree."
if (p_value < alpha) == (p_value_w < alpha) else
"The parametric and non-parametric tests disagree — investigate further.")
Section 5.6 — Two-Sample Independent $t$-Test
When two independent groups are sampled and we want to compare their means:
$$\boxed{t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}}}$$
The formula above is Welch's $t$-test (default in scipy), which does
not assume equal variances. If variances are equal, a pooled variant
is used (equal_var=True).
Degrees of freedom (Welch–Satterthwaite):
$$df = \frac{\left(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}\right)^2}{\frac{(s_1^2/n_1)^2}{n_1-1} + \frac{(s_2^2/n_2)^2}{n_2-1}}$$
Example — Energy Expenditure: Obese vs. Lean. Energy expenditure (MJ) is measured for two independent groups. Is there a difference?
# === Section 5.6: Two-sample independent t-test — Energy expenditure ===
obese = np.array([9.21, 7.53, 7.48, 8.08, 8.09, 10.15, 8.40,
10.88, 6.13, 7.90, 11.51, 12.79])
lean = np.array([7.53, 7.48, 8.08, 8.09, 10.15, 8.40, 10.88,
6.13, 7.90, 11.51])
alpha = 0.05
# Step 1: Check equal-variance assumption (Levene's test)
lev_stat, lev_p = levene(obese, lean)
print(f"Levene's test: F = {lev_stat:.4f}, p = {lev_p:.4f}")
equal_var = lev_p >= alpha
print(f"Equal variances assumed: {equal_var}\n")
# Step 2: Two-sample t-test
t_stat, p_value = ttest_ind(obese, lean, equal_var=equal_var)
print("=== Two-Sample t-Test: Obese vs. Lean ===")
print(f"Obese: n={len(obese)}, x̄={obese.mean():.2f}, s={obese.std(ddof=1):.2f}")
print(f"Lean: n={len(lean)}, x̄={lean.mean():.2f}, s={lean.std(ddof=1):.2f}")
print(f"t = {t_stat:.4f}, p = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
# === Section 5.6 (cont.): Non-parametric alternative — Mann-Whitney U ===
u_stat, u_p = mannwhitneyu(obese, lean, alternative="two-sided")
print("=== Mann-Whitney U Test (non-parametric) ===")
print(f"U = {u_stat:.1f}, p = {u_p:.6f}")
print(f"Decision: {'Reject H₀' if u_p < alpha else 'Fail to reject H₀'}")
Section 5.7 — Paired $t$-Test (Dependent Measures)
When the two measurements are not independent (e.g., the same subject measured before and after treatment), we compute the difference $D_i = X_i - Y_i$ for each pair and test whether $\bar{D}$ differs from 0:
$$\boxed{t = \frac{\bar{D}}{s_D / \sqrt{n}}, \quad df = n - 1}$$
Example — Concrete Strength at 2 vs. 7 Days. Concrete strength (MPa) is measured at 2 days and 7 days for 10 samples. Is there a significant difference at $\alpha = 0.01$?
# === Section 5.7: Paired t-test — Concrete strength ===
day2 = np.array([20.5, 18.8, 22.1, 19.6, 21.3, 17.9, 23.0, 20.2, 18.5, 22.8])
day7 = np.array([25.3, 23.1, 27.8, 24.2, 26.5, 22.4, 28.9, 25.0, 23.6, 27.5])
alpha = 0.01
D = day2 - day7
# scipy.stats.ttest_rel computes the paired t-test directly
t_stat, p_value = ttest_rel(day2, day7)
print("=== Paired t-Test: Concrete Strength (2 days vs. 7 days) ===")
print(f"n = {len(D)} paired observations")
print(f"D̄ = {D.mean():.2f} MPa, s_D = {D.std(ddof=1):.2f} MPa")
print(f"t = {t_stat:.4f}, df = {len(D)-1}, p = {p_value:.8f}")
print(f"\nDecision (α={alpha}): {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
print("Conclusion: Concrete strength at 7 days is significantly greater "
"than at 2 days." if p_value < alpha else
"Insufficient evidence of a difference.")
Section 5.8 — Test of Proportions
For categorical data we test whether a population proportion $p$ equals a hypothesised value $p_0$.
One-Sample Proportion $z$-Test
$$\boxed{z = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}}}$$
Example — Underfilled Bottles. Five years ago 30 % of bottles were underfilled ($p_0 = 0.30$). After corrective measures, 76 of 300 tested bottles are underfilled ($\hat{p} = 0.2533$). Was the correction effective ($\alpha = 0.05$, left-tailed)?
Two-Sample Proportion $z$-Test
$$z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}$$
where $\hat{p} = \frac{x_1 + x_2}{n_1 + n_2}$ is the pooled proportion.
# === Section 5.8: One-sample proportion z-test — Underfilled bottles ===
p0 = 0.30 # historical proportion underfilled
n = 300
x = 76 # underfilled in sample
p_hat = x / n
alpha = 0.05
# Manual calculation
z_stat = (p_hat - p0) / np.sqrt(p0 * (1 - p0) / n)
p_value = norm.cdf(z_stat) # left-tailed
print("=== One-Sample Proportion z-Test: Underfilled Bottles ===")
print(f"H₀: p = {p0} H₁: p < {p0}")
print(f"p̂ = {x}/{n} = {p_hat:.4f}")
print(f"z = {z_stat:.4f}, p-value = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
print("The corrective measure significantly reduced the underfill rate."
if p_value < alpha else "Insufficient evidence.")
# === Section 5.8 (cont.): Two-sample proportion z-test ===
# Agency claims 35% of 155 banks have assets > $10 B.
# Sample: 15 of 50 banks have assets > $10 B.
# Test if proportion < 0.35.
count = np.array([15, 155 * 0.35]) # successes in sample vs. claimed
nobs = np.array([50, 155])
z_stat, p_value = proportions_ztest(count, nobs, alternative="smaller")
print("=== Two-Sample Proportion z-Test: Bank Assets ===")
print(f"Sample: {15}/{50} = {15/50:.2f} Claimed: {0.35:.2f}")
print(f"z = {z_stat:.4f}, p-value = {p_value:.6f}")
print(f"Decision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
Section 5.9 — $\chi^2$ Test of Variance
The chi-square test compares a sample variance to a known population variance. The $\chi^2$ distribution is not symmetric — its shape depends on the degrees of freedom.
$$\boxed{\chi^2 = \frac{(n-1)\,s^2}{\sigma_0^2}, \quad df = n - 1}$$
If $s = \sigma_0$, then $\chi^2 = n - 1$. A much larger $\chi^2$ indicates the sample variance exceeds the population variance.
Example — Coca-Cola Filling Machine. The specification requires $\sigma = 5$ mL. A sample of 10 bottles gives the following volumes:
# === Section 5.9: Chi-square test of variance — Bottle volumes ===
volumes = np.array([607, 602, 590, 603, 598, 602, 597, 600, 603, 598])
sigma_0 = 5.0 # specified population SD
alpha = 0.05
n = len(volumes)
s2 = volumes.var(ddof=1) # sample variance
chi2_stat = (n - 1) * s2 / sigma_0**2
df = n - 1
# Two-tailed critical values
chi2_lo = chi2.ppf(alpha / 2, df)
chi2_hi = chi2.ppf(1 - alpha / 2, df)
# p-value (two-tailed)
p_lo = chi2.cdf(chi2_stat, df)
p_hi = 1 - chi2.cdf(chi2_stat, df)
p_value = 2 * min(p_lo, p_hi)
print("=== χ² Test of Variance: Bottle Filling ===")
print(f"n = {n}, s² = {s2:.2f}, σ₀² = {sigma_0**2:.1f}")
print(f"χ² = (n−1)·s²/σ₀² = {chi2_stat:.4f}, df = {df}")
print(f"Critical region: χ² < {chi2_lo:.4f} or χ² > {chi2_hi:.4f}")
print(f"p-value ≈ {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if (chi2_stat < chi2_lo or chi2_stat > chi2_hi) else 'Fail to reject H₀'}")
Section 5.10 — $F$-Ratio Test (Comparing Two Variances)
The $F$-test compares the variances of two independent samples. The $F$-distribution is right-tailed only (the ratio $s_1^2 / s_2^2$ is always positive).
$$\boxed{F = \frac{s_1^2}{s_2^2}, \quad df_1 = n_1 - 1,\; df_2 = n_2 - 1}$$
Convention: place the larger variance in the numerator.
$H_0$: $\sigma_1^2 = \sigma_2^2$ — the variances are equal.
Example — Stock Volatility. Stock X ($n_1 = 25$) has sample variance $s_1^2 = 6.52$, stock Y ($n_2 = 22$) has $s_2^2 = 3.47$. Are the volatilities different at $\alpha = 0.05$?
# === Section 5.10: F-ratio test — Stock volatility ===
s1_sq, s2_sq = 6.52, 3.47 # sample variances
n1, n2 = 25, 22
alpha = 0.05
F_stat = s1_sq / s2_sq # larger / smaller
df1, df2 = n1 - 1, n2 - 1
# Right-tail p-value (multiply by 2 for two-tailed)
p_value = 2 * (1 - f_dist.cdf(F_stat, df1, df2))
F_crit = f_dist.ppf(1 - alpha / 2, df1, df2) # upper critical value
print("=== F-Ratio Test: Stock Volatility ===")
print(f"s₁² = {s1_sq}, n₁ = {n1} | s₂² = {s2_sq}, n₂ = {n2}")
print(f"F = {s1_sq}/{s2_sq} = {F_stat:.4f}, df₁ = {df1}, df₂ = {df2}")
print(f"F-critical (α/2={alpha/2}) = {F_crit:.4f}")
print(f"p-value = {p_value:.6f}")
print(f"\nDecision: {'Reject H₀' if F_stat > F_crit else 'Fail to reject H₀'} "
f"(variances are {'significantly different' if F_stat > F_crit else 'not significantly different'})")
Section 5.11 — One-Way ANOVA
When comparing the means of three or more independent groups, the $t$-test is insufficient (multiple comparisons inflate Type I error). Analysis of Variance (ANOVA) tests:
$$H_0: \mu_1 = \mu_2 = \cdots = \mu_k \qquad H_a: \text{at least one mean differs}$$
The $F$-statistic for ANOVA is:
$$\boxed{F = \frac{\text{MS}_{\text{between}}}{\text{MS}_{\text{within}}} = \frac{\text{SS}_{\text{between}} \,/\, (k-1)}{\text{SS}_{\text{within}} \,/\, (N-k)}}$$
Assumptions:
- Independent random samples.
- Each group is approximately normally distributed.
- Equal variances across groups (homoscedasticity).
Important: ANOVA tells you that at least one mean differs, but not which one. A post-hoc test (e.g., Tukey HSD) identifies the specific pairs.
Example — Car Crash Head Pressure. Head-pressure readings (arbitrary units) from crashes involving three car types:
# === Section 5.11: One-way ANOVA — Car crash head pressure ===
compact = np.array([643, 655, 702, 685, 671])
midsize = np.array([469, 427, 525, 512, 497])
fullsz = np.array([484, 456, 402, 513, 495])
alpha = 0.05
# Step 1: Assumption checks
for name, grp in [("Compact", compact), ("Midsize", midsize), ("Full-size", fullsz)]:
w, p = shapiro(grp)
print(f"Shapiro-Wilk ({name}): W={w:.4f}, p={p:.4f} "
f"{'✓ Normal' if p >= alpha else '✗ Non-normal'}")
lev_stat, lev_p = levene(compact, midsize, fullsz)
print(f"Levene's test: F={lev_stat:.4f}, p={lev_p:.4f} "
f"{'✓ Equal var' if lev_p >= alpha else '✗ Unequal var'}\n")
# Step 2: One-way ANOVA
F_stat, p_value = f_oneway(compact, midsize, fullsz)
print("=== One-Way ANOVA: Car Crash Head Pressure ===")
print(f"Compact: x̄ = {compact.mean():.1f}")
print(f"Midsize: x̄ = {midsize.mean():.1f}")
print(f"Full-size: x̄ = {fullsz.mean():.1f}")
print(f"F = {F_stat:.4f}, p = {p_value:.8f}")
print(f"\nDecision: {'Reject H₀' if p_value < alpha else 'Fail to reject H₀'}")
print("At least one car type has a significantly different mean head pressure."
if p_value < alpha else "No significant difference detected.")
# === Section 5.11 (cont.): Post-hoc — Tukey HSD ===
# Combine data for Tukey HSD
data = np.concatenate([compact, midsize, fullsz])
labels = (["Compact"] * len(compact) +
["Midsize"] * len(midsize) +
["Full-size"] * len(fullsz))
tukey = pairwise_tukeyhsd(data, labels, alpha=alpha)
print("=== Tukey HSD Post-Hoc Comparison ===")
print(tukey)
print("\n'Reject = True' means the pair has significantly different means.")
# === Section 5.11 (cont.): Boxplot visualisation ===
df_cars = pd.DataFrame({"Pressure": data, "Car Type": labels})
fig, ax = plt.subplots(figsize=(7, 5))
sns.boxplot(data=df_cars, x="Car Type", y="Pressure", palette="Set2", ax=ax)
sns.stripplot(data=df_cars, x="Car Type", y="Pressure", color=".3",
size=6, ax=ax)
ax.set_title(f"ANOVA: Car Crash Head Pressure\n"
f"F = {F_stat:.2f}, p = {p_value:.6f}", fontsize=13)
ax.set_ylabel("Head Pressure (a.u.)")
plt.tight_layout()
plt.show()
Section 5.12 — Assumption Checks
Most parametric tests assume:
- Normality — the data (or residuals) come from a normal distribution.
- Equal variances (homoscedasticity) — required for pooled $t$-test and ANOVA.
| Check | Test | scipy.stats Function |
Null Hypothesis |
|---|---|---|---|
| Normality | Shapiro–Wilk | shapiro(x) |
Data is normally distributed |
| Equal variance | Levene | levene(x1, x2, ...) |
Variances are equal |
If normality is violated → use a non-parametric alternative (§5.13). If equal variance is violated → use Welch's $t$-test or Welch's ANOVA.
# === Section 5.12: Assumption checks — worked example ===
# We reuse the energy-intake data from §5.5 to demonstrate.
intake = np.array([5260, 5470, 5640, 6180, 6390, 6515,
6805, 7515, 7515, 8230, 8770])
# Shapiro-Wilk normality test
w, p_shap = shapiro(intake)
print(f"Shapiro-Wilk: W = {w:.4f}, p = {p_shap:.4f}")
print(f" → {'Normal assumption holds' if p_shap >= 0.05 else 'Normality rejected'}\n")
# Visual: Q-Q plot
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
# Histogram
axes[0].hist(intake, bins=8, edgecolor="white", color="steelblue", alpha=0.7)
axes[0].set_title("Histogram of Energy Intake")
axes[0].set_xlabel("Intake (kJ)")
# Q-Q plot
from scipy.stats import probplot
probplot(intake, dist="norm", plot=axes[1])
axes[1].set_title("Normal Q-Q Plot")
axes[1].get_lines()[0].set_markerfacecolor("steelblue")
axes[1].get_lines()[0].set_markersize(7)
plt.tight_layout()
plt.show()
Section 5.13 — Non-Parametric Alternatives
When the normality assumption fails, use distribution-free tests that operate on ranks rather than raw values.
| Parametric Test | Non-Parametric Alternative | scipy.stats |
|---|---|---|
| One-sample $t$ | Wilcoxon signed-rank | wilcoxon(x - μ₀) |
| Two-sample $t$ (independent) | Mann–Whitney $U$ | mannwhitneyu(x, y) |
| Paired $t$ | Wilcoxon signed-rank | wilcoxon(d) |
| One-way ANOVA | Kruskal–Wallis $H$ | kruskal(x1, x2, ...) |
Non-parametric tests are less powerful (higher $\beta$) when normality actually holds, but they are robust when it does not.
# === Section 5.13: Non-parametric summary — Kruskal-Wallis ===
# Non-parametric alternative to ANOVA using the car crash data from §5.11.
from scipy.stats import kruskal
compact = np.array([643, 655, 702, 685, 671])
midsize = np.array([469, 427, 525, 512, 497])
fullsz = np.array([484, 456, 402, 513, 495])
h_stat, h_p = kruskal(compact, midsize, fullsz)
print("=== Kruskal-Wallis H Test (non-parametric ANOVA) ===")
print(f"H = {h_stat:.4f}, p = {h_p:.6f}")
print(f"Decision: {'Reject H₀' if h_p < 0.05 else 'Fail to reject H₀'}")
print("This confirms the ANOVA result using a distribution-free method.")
Section 5.14 — Summary & Quick Reference
Decision Tree: Which Test?¶
Data type?
├── Categorical ─────── # samples?
│ ├── 1 ──────────── One-proportion z-test
│ ├── 2 ──────────── Two-proportion z-test
│ └── ≥ 2 ─────────── χ² test
│
└── Continuous ────────── σ known?
├── Yes ────────────── z-test
└── No ─────────────── # samples?
├── 1 ──────────── One-sample t-test
├── 2 (independent) ── Two-sample t-test
│ └── equal var? ── Pooled / Welch's
├── 2 (paired) ──── Paired t-test
└── > 2 ──────────── One-way ANOVA
└── post-hoc ── Tukey HSD
Key Formulas¶
| Test | Statistic | Distribution |
|---|---|---|
| $z$-test | $z = \dfrac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}$ | $N(0,1)$ |
| $t$-test | $t = \dfrac{\bar{x} - \mu_0}{s/\sqrt{n}}$ | $t(df=n-1)$ |
| Paired $t$ | $t = \dfrac{\bar{D}}{s_D/\sqrt{n}}$ | $t(df=n-1)$ |
| Proportion $z$ | $z = \dfrac{\hat{p} - p_0}{\sqrt{p_0(1-p_0)/n}}$ | $N(0,1)$ |
| $\chi^2$ variance | $\chi^2 = \dfrac{(n-1)s^2}{\sigma_0^2}$ | $\chi^2(df=n-1)$ |
| $F$-ratio | $F = s_1^2 / s_2^2$ | $F(df_1, df_2)$ |
| ANOVA $F$ | $F = MS_{\text{between}} / MS_{\text{within}}$ | $F(k-1, N-k)$ |
| Confidence interval | $\bar{x} \pm z_{\alpha/2} \cdot \dfrac{\sigma}{\sqrt{n}}$ | — |
Python Cheat Sheet¶
from scipy.stats import (norm, t, chi2, f,
ttest_1samp, ttest_ind, ttest_rel, f_oneway,
shapiro, levene, wilcoxon, mannwhitneyu, kruskal)
from statsmodels.stats.proportion import proportions_ztest
from statsmodels.stats.multicomp import pairwise_tukeyhsd
# One-sample t: ttest_1samp(data, popmean)
# Two-sample t: ttest_ind(a, b, equal_var=False) # Welch's
# Paired t: ttest_rel(before, after)
# Proportion z: proportions_ztest(count, nobs, alternative='smaller')
# ANOVA: f_oneway(g1, g2, g3)
# Tukey HSD: pairwise_tukeyhsd(data, groups)
# Normality: shapiro(x)
# Equal var: levene(a, b)
# Non-parametric: wilcoxon(x - mu) | mannwhitneyu(a, b) | kruskal(g1, g2, g3)
Key Takeaways¶
$$\boxed{\text{If } p < \alpha \text{, reject } H_0. \quad \text{If } p \geq \alpha \text{, fail to reject } H_0.}$$
- "Fail to reject" is not the same as "accept $H_0$".
- Always check assumptions (normality, equal variance) before choosing a parametric test.
- The $p$-value is not the probability that $H_0$ is true — it is the probability of seeing data this extreme if $H_0$ were true.
- Effect size (e.g., Cohen's $d$) complements the $p$-value by quantifying how large the effect is, not just whether it exists.
Next → Chapter 6: Linear Regression
References¶
Wasserman, L. (2004). All of Statistics: A Concise Course in Statistical Inference. Springer. Chapters 10-11 cover the hypothesis-testing framework used in this chapter.
Casella, G., & Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury. Rigorous treatment of test statistics, power, and Type I/II error.
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA's statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129-133. The authoritative statement on correct interpretation and misuse of p-values.
Student's t-test; Welch, B. L. (1947). The generalization of Student's problem when several different population variances are involved. Biometrika, 34(1-2), 28-35. Origin of the unequal-variance t-test used by default in modern statistical software.
SciPy Development Team. scipy.stats hypothesis testing reference (
ttest_ind,chi2_contingency,mannwhitneyu). https://docs.scipy.org/doc/scipy/reference/stats.htmlstatsmodels Development Team. statsmodels documentation (power analysis, multiple comparisons). https://www.statsmodels.org/stable/index.html