Statistical Basis of AI/ML — Regression \& Linear Models
Chapter 6 — Predicting Continuous Outcomes
Why This Chapter?
In Chapters 4 and 5 you learned to quantify uncertainty with probability distributions and to make decisions with hypothesis tests. Now we put those tools to work on one of the most important tasks in data science: predicting a continuous numerical outcome from one or more input features.
Linear regression is the workhorse of predictive modelling. It appears everywhere — pricing models, demand forecasting, clinical dose–response curves, and as the building block inside more complex algorithms (neural networks are, at their core, stacked linear regressions with non-linear activations).
In this chapter you will learn:
- How simple and multiple linear regression work, including the ordinary-least-squares (OLS) objective and its geometric intuition.
- How to evaluate a regression model with $R^2$, Adjusted $R^2$, RMSE, and MAE.
- How to diagnose problems using residual plots and assumption checks.
- How feature scaling and polynomial / interaction features extend the linear framework.
- How Ridge (L2) and Lasso (L1) regularisation tame overfitting and perform feature selection.
- How statsmodels complements scikit-learn by adding inferential statistics (p-values, confidence intervals, F-tests).
- A full case study on the WHO Life Expectancy dataset that ties every technique together.
What You Will Learn
| $\S$ | Topic | Description |
|---|---|---|
| 6.0 | Setup | Imports, GPU check, plot configuration |
| 6.1 | Simple Linear Regression | One predictor, OLS objective, best-fit line |
| 6.2 | Multiple Linear Regression | Many predictors, data prep, train/test split |
| 6.3 | Model Evaluation Metrics | $R^2$, Adjusted $R^2$, RMSE, MAE |
| 6.4 | Residual Analysis & Assumptions | Four OLS assumptions, diagnostic plots |
| 6.5 | Feature Scaling & Polynomial Features | Z-score, interaction terms, complexity trade-off |
| 6.6 | Regularisation — Ridge & Lasso | Bias–variance, L1/L2 penalties, feature selection |
| 6.7 | Inferential Regression with statsmodels | OLS summary, p-values, CI, F-statistic |
| 6.8 | Case Study — Life Expectancy (WHO) | Full EDA → model → diagnostics → recommendations |
| 6.9 | Summary & Quick Reference | Formulas, cheat sheet, key takeaways |
Section 6.0 — Setup: Imports & Configuration
This chapter introduces scikit-learn for model fitting and statsmodels for inferential analysis. The GPU is enabled by default for any PyTorch operations.
| Library | Role in This Chapter |
|---|---|
| NumPy | Array arithmetic, random seeding |
| Pandas | DataFrames, data wrangling |
| Matplotlib | Regression lines, residual plots |
| Seaborn | Pair plots, heat maps, residual plots |
SciPy stats |
Z-score scaling, Shapiro–Wilk |
| scikit-learn | LinearRegression, Ridge, Lasso, PolynomialFeatures, metrics |
| statsmodels | OLS with summary table, formula API |
| PyTorch | GPU availability check |
# === Chapter 6: Setup ===
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import scipy
from scipy import stats
from scipy.stats import zscore, shapiro
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures, scale
from sklearn.metrics import (
mean_squared_error, mean_absolute_error, r2_score,
)
import sklearn
import statsmodels.api as sm
import statsmodels.formula.api as smf
import statsmodels
# ── Plot defaults (house style) ──
plt.rcParams.update({
"figure.figsize": (8, 4),
"axes.grid": True,
"grid.alpha": 0.3,
"axes.spines.top": False,
"axes.spines.right": False,
})
sns.set_theme(style="whitegrid")
rng = np.random.default_rng(42)
print(f"NumPy {np.__version__} | Pandas {pd.__version__} | "
f"SciPy {scipy.__version__}")
print(f"scikit-learn {sklearn.__version__} | "
f"statsmodels {statsmodels.__version__}")
print("\nChapter 6 environment ready.")
Section 6.1 — Simple Linear Regression
The Model¶
Given a single predictor $x$ and a response $y$, simple linear regression fits a straight line:
$$\boxed{\hat{y} = \beta_0 + \beta_1 x}$$
where $\beta_0$ is the intercept (value of $\hat{y}$ when $x = 0$) and $\beta_1$ is the slope (change in $\hat{y}$ per unit increase in $x$).
The OLS Objective¶
The ordinary-least-squares (OLS) method chooses $\beta_0$ and $\beta_1$ to minimise the sum of squared residuals:
$$\boxed{\min_{\beta_0,\,\beta_1}\; \sum_{i=1}^{n} \left(y_i - \hat{y}_i\right)^2 \;=\; \sum_{i=1}^{n}\left(y_i - \beta_0 - \beta_1 x_i\right)^2}$$
Geometrically, OLS finds the line that minimises the total area of the vertical-error squares (hence the name least squares).
We will use the Auto MPG dataset — 398 cars from the 1970s–80s with
fuel efficiency (mpg) as the target.
# === Section 6.1: Load Auto MPG & simple linear regression ===
from pathlib import Path
DATA = Path("../../data")
# Load & clean
df_auto = pd.read_csv(DATA / "auto-mpg.csv")
df_auto["horsepower"] = pd.to_numeric(df_auto["horsepower"], errors="coerce")
df_auto.dropna(subset=["horsepower"], inplace=True)
df_auto.reset_index(drop=True, inplace=True)
print(f"Auto MPG: {df_auto.shape[0]} rows × {df_auto.shape[1]} columns")
df_auto.head()
# === Section 6.1: Scatter plot with best-fit line (weight → mpg) ===
X_simple = df_auto[["weight"]].values
y = df_auto["mpg"].values
model_simple = LinearRegression().fit(X_simple, y)
y_pred_line = model_simple.predict(X_simple)
fig, ax = plt.subplots(figsize=(8, 5))
ax.scatter(X_simple, y, alpha=0.5, s=20, color="steelblue", label="Data")
sort_idx = X_simple[:, 0].argsort()
ax.plot(X_simple[sort_idx, 0], y_pred_line[sort_idx], "r-", lw=2,
label=f"OLS: ŷ = {model_simple.intercept_:.2f} + "
f"({model_simple.coef_[0]:.4f}) × weight")
ax.set_xlabel("Weight (lbs)")
ax.set_ylabel("MPG")
ax.set_title("Simple Linear Regression — Weight vs. MPG")
ax.legend()
plt.tight_layout()
plt.show()
print(f"→ Intercept β₀ = {model_simple.intercept_:.4f}")
print(f"→ Slope β₁ = {model_simple.coef_[0]:.6f}")
print(f"→ R² = {model_simple.score(X_simple, y):.4f}")
print(f"\n→ Interpretation: each additional pound of weight is associated "
f"with a decrease of {abs(model_simple.coef_[0]):.4f} mpg.")
Section 6.2 — Multiple Linear Regression
When we add more predictors the model generalises to:
$$\boxed{\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p}$$
In matrix notation: $\hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\beta}$, where $\mathbf{X}$ is the $n \times (p+1)$ design matrix (with a column of ones for the intercept).
Data Preparation Pipeline¶
Real-world data almost always needs cleaning before regression:
- Handle missing values — impute or drop.
- Encode categorical variables —
pd.get_dummies(one-hot). - Train / test split — hold out data the model has never seen.
We continue with the Auto MPG dataset.
# === Section 6.2: Data preparation & multiple linear regression ===
# One-hot encode 'origin' (1=USA, 2=Europe, 3=Japan)
df_model = df_auto.copy()
df_model = pd.get_dummies(df_model, columns=["origin"], dtype=int)
# Drop non-numeric / identifier columns
drop_cols = ["car name"]
df_model.drop(columns=[c for c in drop_cols if c in df_model.columns],
inplace=True)
# Features & target
target = "mpg"
features = [c for c in df_model.columns if c != target]
X = df_model[features].values
y = df_model[target].values
# 70:30 train/test split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.30, random_state=42
)
# Fit multiple regression
model_multi = LinearRegression().fit(X_train, y_train)
print(f"Training R² = {model_multi.score(X_train, y_train):.4f}")
print(f"Test R² = {model_multi.score(X_test, y_test):.4f}")
print(f"\nCoefficients:")
for name, coef in zip(features, model_multi.coef_):
print(f" {name:20s} β = {coef:+.4f}")
print(f" {'intercept':20s} β₀ = {model_multi.intercept_:+.4f}")
Section 6.3 — Model Evaluation Metrics
$R^2$ — Coefficient of Determination
$$\boxed{R^2 = 1 - \frac{SSE}{SST} = 1 - \frac{\displaystyle\sum_{i=1}^n (y_i - \hat{y}_i)^2}{\displaystyle\sum_{i=1}^n (y_i - \bar{y})^2}}$$
$R^2$ measures the proportion of variance explained by the model. $R^2 = 1$ means perfect prediction; $R^2 = 0$ means the model is no better than predicting the mean.
Adjusted $R^2$
Adding features can only increase $R^2$, even if the new feature is noise. Adjusted $R^2$ penalises model complexity:
$$\boxed{R^2_{\text{adj}} = 1 - \frac{(1 - R^2)(n - 1)}{n - p - 1}}$$
where $n$ is the number of observations and $p$ the number of predictors.
RMSE and MAE¶
$$\boxed{RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^n (y_i - \hat{y}_i)^2}} \qquad \boxed{MAE = \frac{1}{n}\sum_{i=1}^n |y_i - \hat{y}_i|}$$
- RMSE penalises large errors more heavily (quadratic).
- MAE is more robust to outliers (linear).
# === Section 6.3: Evaluation metrics — R², RMSE, MAE ===
y_pred_train = model_multi.predict(X_train)
y_pred_test = model_multi.predict(X_test)
# Custom R² decomposition
SSE = np.sum((y_test - y_pred_test) ** 2)
SST = np.sum((y_test - y_test.mean()) ** 2)
R2_manual = 1 - SSE / SST
n, p = X_test.shape
R2_adj = 1 - (1 - R2_manual) * (n - 1) / (n - p - 1)
rmse_test = np.sqrt(mean_squared_error(y_test, y_pred_test))
mae_test = mean_absolute_error(y_test, y_pred_test)
print("═══ Test-Set Metrics ═══")
print(f" R² = {R2_manual:.4f} (sklearn: {r2_score(y_test, y_pred_test):.4f})")
print(f" Adjusted R² = {R2_adj:.4f}")
print(f" RMSE = {rmse_test:.4f}")
print(f" MAE = {mae_test:.4f}")
print(f"\n SSE = {SSE:.2f} | SST = {SST:.2f}")
# === Section 6.3: Predicted vs. actual scatter plot ===
fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(y_test, y_pred_test, alpha=0.6, s=30, color="steelblue",
edgecolors="white", linewidths=0.5)
lims = [min(y_test.min(), y_pred_test.min()) - 2,
max(y_test.max(), y_pred_test.max()) + 2]
ax.plot(lims, lims, "r--", lw=1.5, label="Perfect prediction")
ax.set_xlim(lims)
ax.set_ylim(lims)
ax.set_xlabel("Actual MPG")
ax.set_ylabel("Predicted MPG")
ax.set_title("Predicted vs. Actual — Multiple Regression (Auto MPG)")
ax.legend()
ax.set_aspect("equal")
plt.tight_layout()
plt.show()
print("→ Points close to the red line indicate good predictions.")
Section 6.4 — Residual Analysis & OLS Assumptions
A residual is $e_i = y_i - \hat{y}_i$. If the model is well-specified the residuals should be:
| Assumption | What It Means | Diagnostic |
|---|---|---|
| Linearity | $E[y \mid x]$ is linear in $x$ | Residual-vs-predicted plot |
| Independence | Residuals are not correlated | Durbin–Watson test |
| Homoscedasticity | $\text{Var}(e_i)$ is constant | Residual spread plot |
| Normality | $e_i \sim \mathcal{N}(0,\,\sigma^2)$ | Q–Q plot, Shapiro–Wilk |
These are the same assumptions from the inferential framework in Chapter 5. Violations do not mean the model is useless — they limit how much we can trust the $p$-values and confidence intervals from statsmodels.
# === Section 6.4: Residual diagnostic plots ===
residuals = y_test - y_pred_test
fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))
# 1. Residual vs. Predicted
axes[0].scatter(y_pred_test, residuals, alpha=0.6, s=25, color="steelblue",
edgecolors="white", linewidths=0.5)
axes[0].axhline(0, color="red", ls="--", lw=1.2)
axes[0].set_xlabel("Predicted MPG")
axes[0].set_ylabel("Residual")
axes[0].set_title("Residuals vs. Predicted")
# 2. Histogram of residuals
axes[1].hist(residuals, bins=20, density=True, alpha=0.6, color="steelblue",
edgecolor="white")
x_norm = np.linspace(residuals.min(), residuals.max(), 200)
axes[1].plot(x_norm, stats.norm.pdf(x_norm, residuals.mean(), residuals.std()),
"r-", lw=2)
axes[1].set_xlabel("Residual")
axes[1].set_ylabel("Density")
axes[1].set_title("Residual Distribution")
# 3. Q–Q plot
stats.probplot(residuals, dist="norm", plot=axes[2])
axes[2].set_title("Normal Q–Q Plot")
axes[2].get_lines()[0].set_color("steelblue")
axes[2].get_lines()[0].set_alpha(0.6)
fig.suptitle("Residual Diagnostics — Multiple Regression (Auto MPG)",
fontsize=13, y=1.02)
plt.tight_layout()
plt.show()
# Shapiro–Wilk normality test on residuals
stat_sw, p_sw = shapiro(residuals)
print(f"Shapiro–Wilk test on residuals: W = {stat_sw:.4f}, p = {p_sw:.4f}")
if p_sw >= 0.05:
print("→ Fail to reject normality (p ≥ 0.05).")
else:
print("→ Residuals depart from normality (p < 0.05) — "
"common with real data; OLS point estimates remain valid "
"but inferential statistics should be interpreted with caution.")
Section 6.5 — Feature Scaling & Polynomial Features
Z-Score Standardisation¶
When features live on very different scales (e.g., weight in thousands of pounds vs. model year 70–82), the coefficients are not directly comparable. Z-score scaling puts every feature on the same footing:
$$\boxed{z = \frac{x - \mu}{\sigma}}$$
After scaling, each $\beta_j$ tells you how many standard deviations of $\hat{y}$ change per one-standard-deviation increase in $x_j$.
Note: Scaling does not change $R^2$ for OLS — it only changes the magnitude and interpretability of the coefficients.
Polynomial & Interaction Features¶
Linear regression is linear in the parameters, not necessarily in $x$. We can add $x^2$, $x_1 x_2$, etc. as new columns:
$$\hat{y} = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x_1 x_2 + \cdots$$
sklearn.preprocessing.PolynomialFeatures automates this — but watch out:
the number of features grows combinatorially.
# === Section 6.5: Z-score scaling + polynomial/interaction features ===
# 1. Z-score scaling
X_train_z = scale(X_train)
X_test_z = scale(X_test)
model_z = LinearRegression().fit(X_train_z, y_train)
print("═══ Standardised Coefficients ═══")
for name, coef in sorted(zip(features, model_z.coef_),
key=lambda x: abs(x[1]), reverse=True):
print(f" {name:20s} β* = {coef:+.4f}")
print(f"\n R² (train) = {model_z.score(X_train_z, y_train):.4f}")
print(f" R² (test) = {model_z.score(X_test_z, y_test):.4f}")
print("\n→ Standardised coefficients rank feature importance by effect size.")
# === Section 6.5 (cont.): Polynomial + interaction features ===
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
X_train_poly = poly.fit_transform(X_train)
X_test_poly = poly.transform(X_test)
model_poly = LinearRegression().fit(X_train_poly, y_train)
print(f"Original features: {X_train.shape[1]}")
print(f"After PolynomialFeatures(degree=2, interaction_only=True): "
f"{X_train_poly.shape[1]}")
print(f"\nTrain R² = {model_poly.score(X_train_poly, y_train):.4f}")
print(f"Test R² = {model_poly.score(X_test_poly, y_test):.4f}")
if model_poly.score(X_test_poly, y_test) < model_multi.score(X_test, y_test):
print("\n→ Test R² decreased — polynomial features overfit on this split.")
else:
improvement = (model_poly.score(X_test_poly, y_test)
- model_multi.score(X_test, y_test))
print(f"\n→ Test R² improved by {improvement:.4f} — "
f"interactions capture non-linear relationships.")
print("\n⚠ Adding interaction terms grew the feature count from "
f"{X_train.shape[1]} to {X_train_poly.shape[1]} — "
"regularisation can help control complexity (see §6.6).")
Section 6.6 — Regularisation: Ridge & Lasso
The Bias–Variance Trade-off¶
OLS minimises training error (bias) but can overfit — especially with many features or collinear predictors (high variance). Regularisation adds a penalty to the OLS objective, trading a small increase in bias for a large decrease in variance.
Ridge Regression (L2 Penalty)¶
$$\boxed{\min_{\boldsymbol{\beta}}\; \sum(y_i - \hat{y}_i)^2 + \alpha \sum_{j=1}^p \beta_j^2}$$
Ridge shrinks coefficients towards zero but never sets them exactly to zero. It works well when many features contribute small effects.
Lasso Regression (L1 Penalty)¶
$$\boxed{\min_{\boldsymbol{\beta}}\; \sum(y_i - \hat{y}_i)^2 + \alpha \sum_{j=1}^p |\beta_j|}$$
Lasso can drive coefficients exactly to zero, performing automatic feature selection. It is preferred when you suspect only a few features matter.
When to Use Which?¶
| Situation | Recommendation |
|---|---|
| Many small-effect features | Ridge |
| Sparse model (few features matter) | Lasso |
| Highly collinear features | Ridge (more stable) |
| Interpretability / feature selection | Lasso |
| Not sure | Try both and compare |
# === Section 6.6: Ridge & Lasso on scaled Auto MPG ===
X_train_s = scale(X_train)
X_test_s = scale(X_test)
models = {
"OLS": LinearRegression().fit(X_train_s, y_train),
"Ridge": Ridge(alpha=1.0).fit(X_train_s, y_train),
"Lasso": Lasso(alpha=0.5).fit(X_train_s, y_train),
}
print("═══ Comparison: OLS vs. Ridge vs. Lasso ═══\n")
# Metric table
print(f"{'Model':8s} {'Train R²':>9s} {'Test R²':>9s} {'# Non-zero':>10s}")
print("─" * 42)
for name, m in models.items():
tr = m.score(X_train_s, y_train)
te = m.score(X_test_s, y_test)
nz = np.sum(np.abs(m.coef_) > 1e-6)
print(f"{name:8s} {tr:9.4f} {te:9.4f} {nz:10d}")
# === Section 6.6 (cont.): Coefficient comparison bar chart ===
coef_df = pd.DataFrame({name: m.coef_ for name, m in models.items()},
index=features)
fig, ax = plt.subplots(figsize=(10, 5))
coef_df.plot(kind="barh", ax=ax, color=["steelblue", "#ef8a62", "#b2df8a"],
edgecolor="white", linewidth=0.5)
ax.axvline(0, color="black", lw=0.8)
ax.set_xlabel("Coefficient Value (standardised)")
ax.set_title("OLS vs. Ridge vs. Lasso — Coefficient Comparison")
ax.legend(frameon=True)
plt.tight_layout()
plt.show()
print("→ Lasso zeros-out features that contribute least,")
print(" acting as an automatic feature-selection mechanism.")
# === Section 6.6 (cont.): Regularisation on polynomial features ===
X_train_poly_s = scale(X_train_poly)
X_test_poly_s = scale(X_test_poly)
poly_models = {
"Poly OLS": LinearRegression().fit(X_train_poly_s, y_train),
"Poly Ridge": Ridge(alpha=1.0).fit(X_train_poly_s, y_train),
"Poly Lasso": Lasso(alpha=0.5).fit(X_train_poly_s, y_train),
}
print("═══ Polynomial Features + Regularisation ═══\n")
print(f"{'Model':14s} {'Train R²':>9s} {'Test R²':>9s} "
f"{'# Non-zero':>10s} {'# Features':>10s}")
print("─" * 60)
for name, m in poly_models.items():
tr = m.score(X_train_poly_s, y_train)
te = m.score(X_test_poly_s, y_test)
nz = np.sum(np.abs(m.coef_) > 1e-6)
print(f"{name:14s} {tr:9.4f} {te:9.4f} "
f"{nz:10d} {X_train_poly_s.shape[1]:10d}")
print("\n→ Regularisation prevents overfitting even with "
f"{X_train_poly_s.shape[1]} polynomial features.")
Section 6.7 — Inferential Regression with statsmodels
scikit-learn excels at prediction but deliberately omits $p$-values and confidence intervals. When you need to know which predictors are statistically significant, use statsmodels.
Two APIs¶
| API | How | When |
|---|---|---|
sm.OLS(y, X_with_const).fit() |
Add constant manually | Full control |
smf.ols("y ~ x1 + x2", data=df).fit() |
R-style formula | Convenience, automatic dummies |
Reading the Summary Table¶
| Field | Meaning |
|---|---|
| Adj. R² | Adjusted coefficient of determination |
| F-statistic | Overall model significance (compare to "Prob (F)") |
| coef | Point estimate $\hat{\beta}_j$ |
| P > |t| | Two-sided $p$-value for $H_0: \beta_j = 0$ |
| [0.025, 0.975] | 95 % confidence interval for $\beta_j$ |
| Durbin–Watson | Autocorrelation of residuals (≈ 2 is ideal) |
Connection to Chapter 5: Each coefficient row in the summary is effectively a $t$-test of $H_0: \beta_j = 0$ — exactly the framework you learned in §5.5.
# === Section 6.7: statsmodels OLS — full summary ===
# Use the formula API for clarity (auto-adds intercept & handles names)
df_sm = df_model.copy()
formula = "mpg ~ " + " + ".join(
[c.replace(" ", "_") for c in features]
)
# Ensure column names match formula
df_sm.columns = [c.replace(" ", "_") for c in df_sm.columns]
ols_result = smf.ols(formula, data=df_sm).fit()
print(ols_result.summary())
# === Section 6.7 (cont.): Extracting key inferential results ===
print("\u2550\u2550\u2550 Significant Predictors (p < 0.05) \u2550\u2550\u2550\n")
sig = ols_result.pvalues[ols_result.pvalues < 0.05].sort_values()
for name, pval in sig.items():
coef = ols_result.params[name]
ci = ols_result.conf_int().loc[name]
print(f" {name:20s} \u03b2 = {coef:+8.4f} "
f"p = {pval:.4f} CI = [{ci[0]:+.4f}, {ci[1]:+.4f}]")
print(f"\n Adj. R\u00b2 = {ols_result.rsquared_adj:.4f}")
print(f" F-statistic = {ols_result.fvalue:.2f} "
f"(p = {ols_result.f_pvalue:.2e})")
print(" Durbin\u2013Watson = see summary table above")
print(f"\n\u2192 sklearn gives prediction; statsmodels gives inference.")
Section 6.8 — Case Study: Life Expectancy (WHO)
We now apply every technique from this chapter to a real-world dataset published by the World Health Organisation — 2938 observations across 193 countries (2000–2015) with 22 attributes covering mortality, immunisation, economic, and social factors.
Target: Life expectancy (years)
This case study follows the complete modelling workflow:
- Exploratory Data Analysis — distributions, correlations, outliers
- Data Preparation — missing values, encoding, train/test split
- Model Fitting — OLS via sklearn, evaluate metrics
- Diagnostics — residual analysis
- Inferential Analysis — statsmodels summary
- Recommendations — which factors most influence life expectancy?
# === Section 6.8: Load WHO Life Expectancy dataset ===
df_life = pd.read_csv(DATA / "Life Expectancy Data.csv")
df_life.columns = df_life.columns.str.strip()
print(f"Shape: {df_life.shape}")
print(f"\nMissing values (top 10):")
print(df_life.isnull().sum().sort_values(ascending=False).head(10))
df_life.head()
# === Section 6.8: EDA — target distribution & correlation heatmap ===
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Target distribution
axes[0].hist(df_life["Life expectancy"].dropna(), bins=40, alpha=0.7,
color="steelblue", edgecolor="white")
axes[0].axvline(df_life["Life expectancy"].mean(), color="red", ls="--",
lw=1.5, label=f"Mean = {df_life['Life expectancy'].mean():.1f}")
axes[0].set_xlabel("Life Expectancy (years)")
axes[0].set_ylabel("Count")
axes[0].set_title("Distribution of Life Expectancy")
axes[0].legend()
# Correlation heatmap (numeric columns only)
numeric_cols = df_life.select_dtypes(include="number")
corr = numeric_cols.corr()
mask = np.triu(np.ones_like(corr, dtype=bool))
sns.heatmap(corr, mask=mask, cmap="RdBu_r", center=0, ax=axes[1],
fmt=".1f", linewidths=0.5, square=True,
cbar_kws={"shrink": 0.8})
axes[1].set_title("Feature Correlation Matrix")
fig.suptitle("WHO Life Expectancy — Exploratory Analysis", fontsize=14,
y=1.02)
plt.tight_layout()
plt.show()
# Top correlations with target
target_corr = corr["Life expectancy"].drop("Life expectancy").sort_values()
print("═══ Top Correlations with Life Expectancy ═══")
print(f"\nStrongest positive: {target_corr.iloc[-1]:.3f} "
f"({target_corr.index[-1]})")
print(f"Strongest negative: {target_corr.iloc[0]:.3f} "
f"({target_corr.index[0]})")
# === Section 6.8: EDA — scatter plots of top predictors ===
top_pos = ["Schooling", "Income composition of resources", "BMI"]
top_neg = ["Adult Mortality", "HIV/AIDS", "Infant deaths"]
fig, axes = plt.subplots(2, 3, figsize=(15, 8))
for ax, col in zip(axes[0], top_pos):
ax.scatter(df_life[col], df_life["Life expectancy"], s=8, alpha=0.3,
color="steelblue")
ax.set_xlabel(col.strip())
ax.set_ylabel("Life Expectancy")
ax.set_title(f"{col.strip()} (positive)")
for ax, col in zip(axes[1], top_neg):
ax.scatter(df_life[col], df_life["Life expectancy"], s=8, alpha=0.3,
color="#ef8a62")
ax.set_xlabel(col.strip())
ax.set_ylabel("Life Expectancy")
ax.set_title(f"{col.strip()} (negative)")
fig.suptitle("Key Predictors of Life Expectancy", fontsize=14, y=1.01)
plt.tight_layout()
plt.show()
# === Section 6.8: Data preparation — clean, encode, split ===
df_le = df_life.copy()
# Drop Country (high-cardinality, not a predictor for a general model)
df_le.drop(columns=["Country"], inplace=True)
# Encode 'Status' (Developing / Developed)
df_le = pd.get_dummies(df_le, columns=["Status"], dtype=int)
# Drop rows with missing target
df_le.dropna(subset=["Life expectancy"], inplace=True)
# Fill remaining NaN with column medians
df_le.fillna(df_le.median(numeric_only=True), inplace=True)
# Features & target
target_le = "Life expectancy"
features_le = [c for c in df_le.columns if c != target_le]
X_le = df_le[features_le].values
y_le = df_le[target_le].values
X_le_train, X_le_test, y_le_train, y_le_test = train_test_split(
X_le, y_le, test_size=0.30, random_state=42
)
print(f"Training set: {X_le_train.shape}")
print(f"Test set: {X_le_test.shape}")
print(f"Features: {len(features_le)}")
# === Section 6.8: Model fitting & evaluation ===
model_le = LinearRegression().fit(X_le_train, y_le_train)
y_le_pred_train = model_le.predict(X_le_train)
y_le_pred_test = model_le.predict(X_le_test)
r2_train = r2_score(y_le_train, y_le_pred_train)
r2_test = r2_score(y_le_test, y_le_pred_test)
rmse_test = np.sqrt(mean_squared_error(y_le_test, y_le_pred_test))
mae_test = mean_absolute_error(y_le_test, y_le_pred_test)
n_te, p_te = X_le_test.shape
r2_adj = 1 - (1 - r2_test) * (n_te - 1) / (n_te - p_te - 1)
print("═══ WHO Life Expectancy — Model Performance ═══\n")
print(f" Training R² = {r2_train:.4f}")
print(f" Test R² = {r2_test:.4f}")
print(f" Adjusted R² = {r2_adj:.4f}")
print(f" RMSE = {rmse_test:.2f} years")
print(f" MAE = {mae_test:.2f} years")
# === Section 6.8: Predicted vs. actual — Life Expectancy ===
fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(y_le_test, y_le_pred_test, alpha=0.4, s=20, color="steelblue",
edgecolors="white", linewidths=0.3)
lims = [35, 92]
ax.plot(lims, lims, "r--", lw=1.5, label="Perfect prediction")
ax.set_xlim(lims)
ax.set_ylim(lims)
ax.set_xlabel("Actual Life Expectancy (years)")
ax.set_ylabel("Predicted Life Expectancy (years)")
ax.set_title("Predicted vs. Actual — WHO Life Expectancy")
ax.legend()
ax.set_aspect("equal")
plt.tight_layout()
plt.show()
# === Section 6.8: Residual diagnostics — Life Expectancy ===
resid_le = y_le_test - y_le_pred_test
fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))
axes[0].scatter(y_le_pred_test, resid_le, alpha=0.4, s=15,
color="steelblue", edgecolors="white", linewidths=0.3)
axes[0].axhline(0, color="red", ls="--", lw=1.2)
axes[0].set_xlabel("Predicted")
axes[0].set_ylabel("Residual")
axes[0].set_title("Residuals vs. Predicted")
axes[1].hist(resid_le, bins=30, density=True, alpha=0.6,
color="steelblue", edgecolor="white")
x_n = np.linspace(resid_le.min(), resid_le.max(), 200)
axes[1].plot(x_n, stats.norm.pdf(x_n, resid_le.mean(), resid_le.std()),
"r-", lw=2)
axes[1].set_xlabel("Residual")
axes[1].set_ylabel("Density")
axes[1].set_title("Residual Distribution")
stats.probplot(resid_le, dist="norm", plot=axes[2])
axes[2].set_title("Normal Q–Q Plot")
axes[2].get_lines()[0].set_color("steelblue")
axes[2].get_lines()[0].set_alpha(0.6)
fig.suptitle("Residual Diagnostics — WHO Life Expectancy", fontsize=13,
y=1.02)
plt.tight_layout()
plt.show()
stat_sw, p_sw = shapiro(resid_le[:500]) # Shapiro limited to ~5000
print(f"Shapiro–Wilk (first 500 residuals): W = {stat_sw:.4f}, p = {p_sw:.4f}")
# === Section 6.8: statsmodels OLS — Life Expectancy ===
# Clean column names for the formula API
df_sm_le = df_le.copy()
df_sm_le.columns = (df_sm_le.columns
.str.strip()
.str.replace(" ", "_")
.str.replace("/", "_")
.str.replace("-", "_"))
target_clean = [c for c in df_sm_le.columns if "life_expectancy" in c.lower()][0]
feat_clean = [c for c in df_sm_le.columns if c != target_clean]
formula_le = f"{target_clean} ~ " + " + ".join(feat_clean)
ols_le = smf.ols(formula_le, data=df_sm_le).fit()
print(ols_le.summary())
# === Section 6.8: Key findings & recommendations ===
print("═══ Key Findings — WHO Life Expectancy ═══\n")
# Top 5 most significant positive & negative predictors
params = ols_le.params.drop("Intercept")
pvals = ols_le.pvalues.drop("Intercept")
sig_mask = pvals < 0.05
sig_params = params[sig_mask].sort_values()
print("Most significant NEGATIVE predictors (lower LE):")
for name, coef in sig_params.head(3).items():
print(f" → {name}: β = {coef:+.4f} (p = {pvals[name]:.4f})")
print("\nMost significant POSITIVE predictors (higher LE):")
for name, coef in sig_params.tail(3).items():
print(f" → {name}: β = {coef:+.4f} (p = {pvals[name]:.4f})")
print("\n═══ Recommendations ═══")
print("→ HIV/AIDS prevalence and adult mortality are the strongest")
print(" negative drivers — public health interventions in these areas")
print(" have the greatest potential to improve life expectancy.")
print("→ Schooling, income composition, and immunisation coverage are")
print(" the strongest positive drivers — investment in education and")
print(" healthcare access yields measurable improvements.")
print(f"\n Overall model: Adj. R² = {ols_le.rsquared_adj:.4f}")
Section 6.9 — Summary & Quick Reference
Key Formulas¶
| Quantity | Formula | |:---------|:--------| | Simple regression | $\hat{y} = \beta_0 + \beta_1 x$ | | Multiple regression | $\hat{y} = \beta_0 + \sum_{j=1}^p \beta_j x_j$ | | OLS objective | $\min \sum(y_i - \hat{y}_i)^2$ | | $R^2$ | $1 - SSE/SST$ | | Adjusted $R^2$ | $1 - \dfrac{(1-R^2)(n-1)}{n-p-1}$ | | RMSE | $\sqrt{\dfrac{1}{n}\sum(y_i-\hat{y}_i)^2}$ | | MAE | $\dfrac{1}{n}\sum|y_i-\hat{y}_i|$ | | Ridge penalty | $+ \alpha \sum \beta_j^2$ | | Lasso penalty | $+ \alpha \sum |\beta_j|$ | | Z-score | $z = (x - \mu) / \sigma$ |
Python Cheat Sheet¶
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures, scale
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error
import statsmodels.formula.api as smf
# OLS (sklearn): model = LinearRegression().fit(X_train, y_train)
# Ridge: model = Ridge(alpha=1.0).fit(X_train, y_train)
# Lasso: model = Lasso(alpha=0.5).fit(X_train, y_train)
# Polynomial: X_poly = PolynomialFeatures(2).fit_transform(X)
# Metrics: r2_score(y, ŷ) | mean_squared_error(y, ŷ)
# statsmodels: result = smf.ols("y ~ x1 + x2", data=df).fit()
# Summary: result.summary()
# Significant: result.pvalues[result.pvalues < 0.05]
Key Takeaways¶
$$\boxed{R^2 \text{ measures fit, Adjusted } R^2 \text{ penalises complexity, RMSE penalises large errors.}}$$
- OLS is the foundation — understand it before moving to regularised methods.
- Residual analysis is not optional: it reveals violated assumptions that invalidate inferences.
- Ridge and Lasso are essential when you have many features or collinear predictors.
- statsmodels complements sklearn by providing hypothesis tests on individual coefficients — connecting this chapter back to Chapter 5.
- The case study showed that real-world regression involves extensive
EDA, data cleaning, and domain-informed interpretation — not just
calling
.fit().
Chapter 2 gave you correlation; Chapter 5 gave you hypothesis tests on parameters; this chapter combined them into a complete predictive framework.
Next → Chapter 7: Classification & Model Selection
References¶
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.). Springer. Chapter 3 covers linear regression at the level used here. https://www.statlearning.com/
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. Chapter 3 provides the graduate-level treatment of least squares and regularization (ridge, lasso). https://hastie.su.domains/ElemStatLearn/
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B, 58(1), 267-288. The original lasso paper referenced by the regularization section.
scikit-learn Developers. scikit-learn
linear_modeldocumentation. https://scikit-learn.org/stable/modules/linear_model.htmlstatsmodels Development Team. statsmodels OLS documentation (diagnostics, VIF, residual plots). https://www.statsmodels.org/stable/regression.html