Home / About Us / Volume I Contents / Ch 06: Regression and Linear Models

Statistical Basis of AI/ML — Regression \& Linear Models

Chapter 6 — Predicting Continuous Outcomes


Why This Chapter?

In Chapters 4 and 5 you learned to quantify uncertainty with probability distributions and to make decisions with hypothesis tests. Now we put those tools to work on one of the most important tasks in data science: predicting a continuous numerical outcome from one or more input features.

Linear regression is the workhorse of predictive modelling. It appears everywhere — pricing models, demand forecasting, clinical dose–response curves, and as the building block inside more complex algorithms (neural networks are, at their core, stacked linear regressions with non-linear activations).

In this chapter you will learn:

  • How simple and multiple linear regression work, including the ordinary-least-squares (OLS) objective and its geometric intuition.
  • How to evaluate a regression model with $R^2$, Adjusted $R^2$, RMSE, and MAE.
  • How to diagnose problems using residual plots and assumption checks.
  • How feature scaling and polynomial / interaction features extend the linear framework.
  • How Ridge (L2) and Lasso (L1) regularisation tame overfitting and perform feature selection.
  • How statsmodels complements scikit-learn by adding inferential statistics (p-values, confidence intervals, F-tests).
  • A full case study on the WHO Life Expectancy dataset that ties every technique together.

What You Will Learn

$\S$ Topic Description
6.0 Setup Imports, GPU check, plot configuration
6.1 Simple Linear Regression One predictor, OLS objective, best-fit line
6.2 Multiple Linear Regression Many predictors, data prep, train/test split
6.3 Model Evaluation Metrics $R^2$, Adjusted $R^2$, RMSE, MAE
6.4 Residual Analysis & Assumptions Four OLS assumptions, diagnostic plots
6.5 Feature Scaling & Polynomial Features Z-score, interaction terms, complexity trade-off
6.6 Regularisation — Ridge & Lasso Bias–variance, L1/L2 penalties, feature selection
6.7 Inferential Regression with statsmodels OLS summary, p-values, CI, F-statistic
6.8 Case Study — Life Expectancy (WHO) Full EDA → model → diagnostics → recommendations
6.9 Summary & Quick Reference Formulas, cheat sheet, key takeaways

Section 6.0 — Setup: Imports & Configuration

This chapter introduces scikit-learn for model fitting and statsmodels for inferential analysis. The GPU is enabled by default for any PyTorch operations.

Library Role in This Chapter
NumPy Array arithmetic, random seeding
Pandas DataFrames, data wrangling
Matplotlib Regression lines, residual plots
Seaborn Pair plots, heat maps, residual plots
SciPy stats Z-score scaling, Shapiro–Wilk
scikit-learn LinearRegression, Ridge, Lasso, PolynomialFeatures, metrics
statsmodels OLS with summary table, formula API
PyTorch GPU availability check
In [1]:
# === Chapter 6: Setup ===

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import scipy
from scipy import stats
from scipy.stats import zscore, shapiro

from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures, scale
from sklearn.metrics import (
    mean_squared_error, mean_absolute_error, r2_score,
)
import sklearn

import statsmodels.api as sm
import statsmodels.formula.api as smf
import statsmodels

# ── Plot defaults (house style) ──
plt.rcParams.update({
    "figure.figsize": (8, 4),
    "axes.grid": True,
    "grid.alpha": 0.3,
    "axes.spines.top": False,
    "axes.spines.right": False,
})
sns.set_theme(style="whitegrid")

rng = np.random.default_rng(42)

print(f"NumPy {np.__version__}  |  Pandas {pd.__version__}  |  "
      f"SciPy {scipy.__version__}")
print(f"scikit-learn {sklearn.__version__}  |  "
      f"statsmodels {statsmodels.__version__}")
print("\nChapter 6 environment ready.")
NumPy 2.5.1  |  Pandas 2.3.3  |  SciPy 1.18.0
scikit-learn 1.9.0  |  statsmodels 0.14.6

Chapter 6 environment ready.

Section 6.1 — Simple Linear Regression

The Model

Given a single predictor $x$ and a response $y$, simple linear regression fits a straight line:

$$\boxed{\hat{y} = \beta_0 + \beta_1 x}$$

where $\beta_0$ is the intercept (value of $\hat{y}$ when $x = 0$) and $\beta_1$ is the slope (change in $\hat{y}$ per unit increase in $x$).

The OLS Objective

The ordinary-least-squares (OLS) method chooses $\beta_0$ and $\beta_1$ to minimise the sum of squared residuals:

$$\boxed{\min_{\beta_0,\,\beta_1}\; \sum_{i=1}^{n} \left(y_i - \hat{y}_i\right)^2 \;=\; \sum_{i=1}^{n}\left(y_i - \beta_0 - \beta_1 x_i\right)^2}$$

Geometrically, OLS finds the line that minimises the total area of the vertical-error squares (hence the name least squares).

We will use the Auto MPG dataset — 398 cars from the 1970s–80s with fuel efficiency (mpg) as the target.

In [2]:
# === Section 6.1: Load Auto MPG & simple linear regression ===

from pathlib import Path
DATA = Path("../../data")

# Load & clean
df_auto = pd.read_csv(DATA / "auto-mpg.csv")
df_auto["horsepower"] = pd.to_numeric(df_auto["horsepower"], errors="coerce")
df_auto.dropna(subset=["horsepower"], inplace=True)
df_auto.reset_index(drop=True, inplace=True)
print(f"Auto MPG: {df_auto.shape[0]} rows × {df_auto.shape[1]} columns")
df_auto.head()
Auto MPG: 392 rows × 9 columns
Out[2]:
mpg cylinders displacement horsepower weight acceleration model year origin car name
0 18.0 8 307.0 130.0 3504 12.0 70 1 chevrolet chevelle malibu
1 15.0 8 350.0 165.0 3693 11.5 70 1 buick skylark 320
2 18.0 8 318.0 150.0 3436 11.0 70 1 plymouth satellite
3 16.0 8 304.0 150.0 3433 12.0 70 1 amc rebel sst
4 17.0 8 302.0 140.0 3449 10.5 70 1 ford torino
In [3]:
# === Section 6.1: Scatter plot with best-fit line (weight → mpg) ===

X_simple = df_auto[["weight"]].values
y = df_auto["mpg"].values

model_simple = LinearRegression().fit(X_simple, y)
y_pred_line = model_simple.predict(X_simple)

fig, ax = plt.subplots(figsize=(8, 5))
ax.scatter(X_simple, y, alpha=0.5, s=20, color="steelblue", label="Data")
sort_idx = X_simple[:, 0].argsort()
ax.plot(X_simple[sort_idx, 0], y_pred_line[sort_idx], "r-", lw=2,
        label=f"OLS: ŷ = {model_simple.intercept_:.2f} + "
              f"({model_simple.coef_[0]:.4f}) × weight")
ax.set_xlabel("Weight (lbs)")
ax.set_ylabel("MPG")
ax.set_title("Simple Linear Regression — Weight vs. MPG")
ax.legend()
plt.tight_layout()
plt.show()

print(f"→ Intercept β₀ = {model_simple.intercept_:.4f}")
print(f"→ Slope     β₁ = {model_simple.coef_[0]:.6f}")
print(f"→ R²           = {model_simple.score(X_simple, y):.4f}")
print(f"\n→ Interpretation: each additional pound of weight is associated "
      f"with a decrease of {abs(model_simple.coef_[0]):.4f} mpg.")
No description has been provided for this image
→ Intercept β₀ = 46.2165
→ Slope     β₁ = -0.007647
→ R²           = 0.6926

→ Interpretation: each additional pound of weight is associated with a decrease of 0.0076 mpg.

Section 6.2 — Multiple Linear Regression

When we add more predictors the model generalises to:

$$\boxed{\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p}$$

In matrix notation: $\hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\beta}$, where $\mathbf{X}$ is the $n \times (p+1)$ design matrix (with a column of ones for the intercept).

Data Preparation Pipeline

Real-world data almost always needs cleaning before regression:

  1. Handle missing values — impute or drop.
  2. Encode categorical variables — pd.get_dummies (one-hot).
  3. Train / test split — hold out data the model has never seen.

We continue with the Auto MPG dataset.

In [4]:
# === Section 6.2: Data preparation & multiple linear regression ===

# One-hot encode 'origin' (1=USA, 2=Europe, 3=Japan)
df_model = df_auto.copy()
df_model = pd.get_dummies(df_model, columns=["origin"], dtype=int)

# Drop non-numeric / identifier columns
drop_cols = ["car name"]
df_model.drop(columns=[c for c in drop_cols if c in df_model.columns],
              inplace=True)

# Features & target
target = "mpg"
features = [c for c in df_model.columns if c != target]
X = df_model[features].values
y = df_model[target].values

# 70:30 train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, random_state=42
)

# Fit multiple regression
model_multi = LinearRegression().fit(X_train, y_train)

print(f"Training R² = {model_multi.score(X_train, y_train):.4f}")
print(f"Test     R² = {model_multi.score(X_test, y_test):.4f}")
print(f"\nCoefficients:")
for name, coef in zip(features, model_multi.coef_):
    print(f"  {name:20s}  β = {coef:+.4f}")
print(f"  {'intercept':20s}  β₀ = {model_multi.intercept_:+.4f}")
Training R² = 0.8265
Test     R² = 0.8045

Coefficients:
  cylinders             β = -0.3331
  displacement          β = +0.0204
  horsepower            β = -0.0288
  weight                β = -0.0063
  acceleration          β = +0.0413
  model year            β = +0.7782
  origin_1              β = -2.2330
  origin_2              β = +1.1039
  origin_3              β = +1.1292
  intercept             β₀ = -15.4999

Section 6.3 — Model Evaluation Metrics

$R^2$ — Coefficient of Determination

$$\boxed{R^2 = 1 - \frac{SSE}{SST} = 1 - \frac{\displaystyle\sum_{i=1}^n (y_i - \hat{y}_i)^2}{\displaystyle\sum_{i=1}^n (y_i - \bar{y})^2}}$$

$R^2$ measures the proportion of variance explained by the model. $R^2 = 1$ means perfect prediction; $R^2 = 0$ means the model is no better than predicting the mean.

Adjusted $R^2$

Adding features can only increase $R^2$, even if the new feature is noise. Adjusted $R^2$ penalises model complexity:

$$\boxed{R^2_{\text{adj}} = 1 - \frac{(1 - R^2)(n - 1)}{n - p - 1}}$$

where $n$ is the number of observations and $p$ the number of predictors.

RMSE and MAE

$$\boxed{RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^n (y_i - \hat{y}_i)^2}} \qquad \boxed{MAE = \frac{1}{n}\sum_{i=1}^n |y_i - \hat{y}_i|}$$

  • RMSE penalises large errors more heavily (quadratic).
  • MAE is more robust to outliers (linear).
In [5]:
# === Section 6.3: Evaluation metrics — R², RMSE, MAE ===

y_pred_train = model_multi.predict(X_train)
y_pred_test  = model_multi.predict(X_test)

# Custom R² decomposition
SSE = np.sum((y_test - y_pred_test) ** 2)
SST = np.sum((y_test - y_test.mean()) ** 2)
R2_manual = 1 - SSE / SST

n, p = X_test.shape
R2_adj = 1 - (1 - R2_manual) * (n - 1) / (n - p - 1)

rmse_test = np.sqrt(mean_squared_error(y_test, y_pred_test))
mae_test  = mean_absolute_error(y_test, y_pred_test)

print("═══ Test-Set Metrics ═══")
print(f"  R²           = {R2_manual:.4f}  (sklearn: {r2_score(y_test, y_pred_test):.4f})")
print(f"  Adjusted R²  = {R2_adj:.4f}")
print(f"  RMSE         = {rmse_test:.4f}")
print(f"  MAE          = {mae_test:.4f}")
print(f"\n  SSE = {SSE:.2f}   |   SST = {SST:.2f}")
═══ Test-Set Metrics ═══
  R²           = 0.8045  (sklearn: 0.8045)
  Adjusted R²  = 0.7882
  RMSE         = 3.2162
  MAE          = 2.5345

  SSE = 1220.61   |   SST = 6242.77
In [6]:
# === Section 6.3: Predicted vs. actual scatter plot ===

fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(y_test, y_pred_test, alpha=0.6, s=30, color="steelblue",
           edgecolors="white", linewidths=0.5)
lims = [min(y_test.min(), y_pred_test.min()) - 2,
        max(y_test.max(), y_pred_test.max()) + 2]
ax.plot(lims, lims, "r--", lw=1.5, label="Perfect prediction")
ax.set_xlim(lims)
ax.set_ylim(lims)
ax.set_xlabel("Actual MPG")
ax.set_ylabel("Predicted MPG")
ax.set_title("Predicted vs. Actual — Multiple Regression (Auto MPG)")
ax.legend()
ax.set_aspect("equal")
plt.tight_layout()
plt.show()

print("→ Points close to the red line indicate good predictions.")
No description has been provided for this image
→ Points close to the red line indicate good predictions.

Section 6.4 — Residual Analysis & OLS Assumptions

A residual is $e_i = y_i - \hat{y}_i$. If the model is well-specified the residuals should be:

Assumption What It Means Diagnostic
Linearity $E[y \mid x]$ is linear in $x$ Residual-vs-predicted plot
Independence Residuals are not correlated Durbin–Watson test
Homoscedasticity $\text{Var}(e_i)$ is constant Residual spread plot
Normality $e_i \sim \mathcal{N}(0,\,\sigma^2)$ Q–Q plot, Shapiro–Wilk

These are the same assumptions from the inferential framework in Chapter 5. Violations do not mean the model is useless — they limit how much we can trust the $p$-values and confidence intervals from statsmodels.

In [7]:
# === Section 6.4: Residual diagnostic plots ===

residuals = y_test - y_pred_test

fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))

# 1. Residual vs. Predicted
axes[0].scatter(y_pred_test, residuals, alpha=0.6, s=25, color="steelblue",
                edgecolors="white", linewidths=0.5)
axes[0].axhline(0, color="red", ls="--", lw=1.2)
axes[0].set_xlabel("Predicted MPG")
axes[0].set_ylabel("Residual")
axes[0].set_title("Residuals vs. Predicted")

# 2. Histogram of residuals
axes[1].hist(residuals, bins=20, density=True, alpha=0.6, color="steelblue",
             edgecolor="white")
x_norm = np.linspace(residuals.min(), residuals.max(), 200)
axes[1].plot(x_norm, stats.norm.pdf(x_norm, residuals.mean(), residuals.std()),
             "r-", lw=2)
axes[1].set_xlabel("Residual")
axes[1].set_ylabel("Density")
axes[1].set_title("Residual Distribution")

# 3. Q–Q plot
stats.probplot(residuals, dist="norm", plot=axes[2])
axes[2].set_title("Normal Q–Q Plot")
axes[2].get_lines()[0].set_color("steelblue")
axes[2].get_lines()[0].set_alpha(0.6)

fig.suptitle("Residual Diagnostics — Multiple Regression (Auto MPG)",
             fontsize=13, y=1.02)
plt.tight_layout()
plt.show()

# Shapiro–Wilk normality test on residuals
stat_sw, p_sw = shapiro(residuals)
print(f"Shapiro–Wilk test on residuals: W = {stat_sw:.4f}, p = {p_sw:.4f}")
if p_sw >= 0.05:
    print("→ Fail to reject normality (p ≥ 0.05).")
else:
    print("→ Residuals depart from normality (p < 0.05) — "
          "common with real data; OLS point estimates remain valid "
          "but inferential statistics should be interpreted with caution.")
No description has been provided for this image
Shapiro–Wilk test on residuals: W = 0.9945, p = 0.9263
→ Fail to reject normality (p ≥ 0.05).

Section 6.5 — Feature Scaling & Polynomial Features

Z-Score Standardisation

When features live on very different scales (e.g., weight in thousands of pounds vs. model year 70–82), the coefficients are not directly comparable. Z-score scaling puts every feature on the same footing:

$$\boxed{z = \frac{x - \mu}{\sigma}}$$

After scaling, each $\beta_j$ tells you how many standard deviations of $\hat{y}$ change per one-standard-deviation increase in $x_j$.

Note: Scaling does not change $R^2$ for OLS — it only changes the magnitude and interpretability of the coefficients.

Polynomial & Interaction Features

Linear regression is linear in the parameters, not necessarily in $x$. We can add $x^2$, $x_1 x_2$, etc. as new columns:

$$\hat{y} = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x_1 x_2 + \cdots$$

sklearn.preprocessing.PolynomialFeatures automates this — but watch out: the number of features grows combinatorially.

In [8]:
# === Section 6.5: Z-score scaling + polynomial/interaction features ===

# 1. Z-score scaling
X_train_z = scale(X_train)
X_test_z  = scale(X_test)

model_z = LinearRegression().fit(X_train_z, y_train)
print("═══ Standardised Coefficients ═══")
for name, coef in sorted(zip(features, model_z.coef_),
                          key=lambda x: abs(x[1]), reverse=True):
    print(f"  {name:20s}  β* = {coef:+.4f}")
print(f"\n  R² (train) = {model_z.score(X_train_z, y_train):.4f}")
print(f"  R² (test)  = {model_z.score(X_test_z, y_test):.4f}")
print("\n→ Standardised coefficients rank feature importance by effect size.")
═══ Standardised Coefficients ═══
  weight                β* = -5.2942
  model year            β* = +2.8116
  displacement          β* = +2.1214
  horsepower            β* = -1.0951
  origin_1              β* = -0.8953
  origin_3              β* = +0.6048
  cylinders             β* = -0.5708
  origin_2              β* = +0.5172
  acceleration          β* = +0.1178

  R² (train) = 0.8265
  R² (test)  = 0.8122

→ Standardised coefficients rank feature importance by effect size.
In [9]:
# === Section 6.5 (cont.): Polynomial + interaction features ===

poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
X_train_poly = poly.fit_transform(X_train)
X_test_poly  = poly.transform(X_test)

model_poly = LinearRegression().fit(X_train_poly, y_train)

print(f"Original features: {X_train.shape[1]}")
print(f"After PolynomialFeatures(degree=2, interaction_only=True): "
      f"{X_train_poly.shape[1]}")
print(f"\nTrain R² = {model_poly.score(X_train_poly, y_train):.4f}")
print(f"Test  R² = {model_poly.score(X_test_poly, y_test):.4f}")

if model_poly.score(X_test_poly, y_test) < model_multi.score(X_test, y_test):
    print("\n→ Test R² decreased — polynomial features overfit on this split.")
else:
    improvement = (model_poly.score(X_test_poly, y_test)
                   - model_multi.score(X_test, y_test))
    print(f"\n→ Test R² improved by {improvement:.4f} — "
          f"interactions capture non-linear relationships.")
print("\n⚠ Adding interaction terms grew the feature count from "
      f"{X_train.shape[1]} to {X_train_poly.shape[1]} — "
      "regularisation can help control complexity (see §6.6).")
Original features: 9
After PolynomialFeatures(degree=2, interaction_only=True): 45

Train R² = 0.8926
Test  R² = 0.8358

→ Test R² improved by 0.0314 — interactions capture non-linear relationships.

⚠ Adding interaction terms grew the feature count from 9 to 45 — regularisation can help control complexity (see §6.6).

Section 6.6 — Regularisation: Ridge & Lasso

The Bias–Variance Trade-off

OLS minimises training error (bias) but can overfit — especially with many features or collinear predictors (high variance). Regularisation adds a penalty to the OLS objective, trading a small increase in bias for a large decrease in variance.

Ridge Regression (L2 Penalty)

$$\boxed{\min_{\boldsymbol{\beta}}\; \sum(y_i - \hat{y}_i)^2 + \alpha \sum_{j=1}^p \beta_j^2}$$

Ridge shrinks coefficients towards zero but never sets them exactly to zero. It works well when many features contribute small effects.

Lasso Regression (L1 Penalty)

$$\boxed{\min_{\boldsymbol{\beta}}\; \sum(y_i - \hat{y}_i)^2 + \alpha \sum_{j=1}^p |\beta_j|}$$

Lasso can drive coefficients exactly to zero, performing automatic feature selection. It is preferred when you suspect only a few features matter.

When to Use Which?

Situation Recommendation
Many small-effect features Ridge
Sparse model (few features matter) Lasso
Highly collinear features Ridge (more stable)
Interpretability / feature selection Lasso
Not sure Try both and compare
In [10]:
# === Section 6.6: Ridge & Lasso on scaled Auto MPG ===

X_train_s = scale(X_train)
X_test_s  = scale(X_test)

models = {
    "OLS":   LinearRegression().fit(X_train_s, y_train),
    "Ridge": Ridge(alpha=1.0).fit(X_train_s, y_train),
    "Lasso": Lasso(alpha=0.5).fit(X_train_s, y_train),
}

print("═══ Comparison: OLS vs. Ridge vs. Lasso ═══\n")

# Metric table
print(f"{'Model':8s}  {'Train R²':>9s}  {'Test R²':>9s}  {'# Non-zero':>10s}")
print("─" * 42)
for name, m in models.items():
    tr = m.score(X_train_s, y_train)
    te = m.score(X_test_s, y_test)
    nz = np.sum(np.abs(m.coef_) > 1e-6)
    print(f"{name:8s}  {tr:9.4f}  {te:9.4f}  {nz:10d}")
═══ Comparison: OLS vs. Ridge vs. Lasso ═══

Model      Train R²    Test R²  # Non-zero
──────────────────────────────────────────
OLS          0.8265     0.8122           9
Ridge        0.8264     0.8111           9
Lasso        0.8158     0.8134           4
In [11]:
# === Section 6.6 (cont.): Coefficient comparison bar chart ===

coef_df = pd.DataFrame({name: m.coef_ for name, m in models.items()},
                        index=features)

fig, ax = plt.subplots(figsize=(10, 5))
coef_df.plot(kind="barh", ax=ax, color=["steelblue", "#ef8a62", "#b2df8a"],
             edgecolor="white", linewidth=0.5)
ax.axvline(0, color="black", lw=0.8)
ax.set_xlabel("Coefficient Value (standardised)")
ax.set_title("OLS vs. Ridge vs. Lasso — Coefficient Comparison")
ax.legend(frameon=True)
plt.tight_layout()
plt.show()

print("→ Lasso zeros-out features that contribute least,")
print("  acting as an automatic feature-selection mechanism.")
No description has been provided for this image
→ Lasso zeros-out features that contribute least,
  acting as an automatic feature-selection mechanism.
In [12]:
# === Section 6.6 (cont.): Regularisation on polynomial features ===

X_train_poly_s = scale(X_train_poly)
X_test_poly_s  = scale(X_test_poly)

poly_models = {
    "Poly OLS":   LinearRegression().fit(X_train_poly_s, y_train),
    "Poly Ridge": Ridge(alpha=1.0).fit(X_train_poly_s, y_train),
    "Poly Lasso": Lasso(alpha=0.5).fit(X_train_poly_s, y_train),
}

print("═══ Polynomial Features + Regularisation ═══\n")
print(f"{'Model':14s}  {'Train R²':>9s}  {'Test R²':>9s}  "
      f"{'# Non-zero':>10s}  {'# Features':>10s}")
print("─" * 60)
for name, m in poly_models.items():
    tr = m.score(X_train_poly_s, y_train)
    te = m.score(X_test_poly_s, y_test)
    nz = np.sum(np.abs(m.coef_) > 1e-6)
    print(f"{name:14s}  {tr:9.4f}  {te:9.4f}  "
          f"{nz:10d}  {X_train_poly_s.shape[1]:10d}")

print("\n→ Regularisation prevents overfitting even with "
      f"{X_train_poly_s.shape[1]} polynomial features.")
═══ Polynomial Features + Regularisation ═══

Model            Train R²    Test R²  # Non-zero  # Features
────────────────────────────────────────────────────────────
Poly OLS           0.9050     0.8402          42          45
Poly Ridge         0.8933     0.8471          42          45
Poly Lasso         0.8305     0.8253           8          45

→ Regularisation prevents overfitting even with 45 polynomial features.

Section 6.7 — Inferential Regression with statsmodels

scikit-learn excels at prediction but deliberately omits $p$-values and confidence intervals. When you need to know which predictors are statistically significant, use statsmodels.

Two APIs

API How When
sm.OLS(y, X_with_const).fit() Add constant manually Full control
smf.ols("y ~ x1 + x2", data=df).fit() R-style formula Convenience, automatic dummies

Reading the Summary Table

Field Meaning
Adj. R² Adjusted coefficient of determination
F-statistic Overall model significance (compare to "Prob (F)")
coef Point estimate $\hat{\beta}_j$
P > |t| Two-sided $p$-value for $H_0: \beta_j = 0$
[0.025, 0.975] 95 % confidence interval for $\beta_j$
Durbin–Watson Autocorrelation of residuals (≈ 2 is ideal)

Connection to Chapter 5: Each coefficient row in the summary is effectively a $t$-test of $H_0: \beta_j = 0$ — exactly the framework you learned in §5.5.

In [13]:
# === Section 6.7: statsmodels OLS — full summary ===

# Use the formula API for clarity (auto-adds intercept & handles names)
df_sm = df_model.copy()
formula = "mpg ~ " + " + ".join(
    [c.replace(" ", "_") for c in features]
)
# Ensure column names match formula
df_sm.columns = [c.replace(" ", "_") for c in df_sm.columns]

ols_result = smf.ols(formula, data=df_sm).fit()
print(ols_result.summary())
                            OLS Regression Results                            
==============================================================================
Dep. Variable:                    mpg   R-squared:                       0.824
Model:                            OLS   Adj. R-squared:                  0.821
Method:                 Least Squares   F-statistic:                     224.5
Date:                Sat, 08 Aug 2026   Prob (F-statistic):          1.79e-139
Time:                        12:36:35   Log-Likelihood:                -1020.5
No. Observations:                 392   AIC:                             2059.
Df Residuals:                     383   BIC:                             2095.
Df Model:                           8                                         
Covariance Type:            nonrobust                                         
================================================================================
                   coef    std err          t      P>|t|      [0.025      0.975]
--------------------------------------------------------------------------------
Intercept      -12.0951      3.482     -3.474      0.001     -18.941      -5.250
cylinders       -0.4897      0.321     -1.524      0.128      -1.121       0.142
displacement     0.0240      0.008      3.133      0.002       0.009       0.039
horsepower      -0.0182      0.014     -1.326      0.185      -0.045       0.009
weight          -0.0067      0.001    -10.243      0.000      -0.008      -0.005
acceleration     0.0791      0.098      0.805      0.421      -0.114       0.272
model_year       0.7770      0.052     15.005      0.000       0.675       0.879
origin_1        -5.8595      1.227     -4.775      0.000      -8.272      -3.447
origin_2        -3.2295      1.156     -2.794      0.005      -5.502      -0.957
origin_3        -3.0062      1.231     -2.443      0.015      -5.426      -0.587
==============================================================================
Omnibus:                       23.395   Durbin-Watson:                   1.291
Prob(Omnibus):                  0.000   Jarque-Bera (JB):               34.452
Skew:                           0.444   Prob(JB):                     3.30e-08
Kurtosis:                       4.150   Cond. No.                     5.35e+18
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The smallest eigenvalue is 1.32e-28. This might indicate that there are
strong multicollinearity problems or that the design matrix is singular.
In [14]:
# === Section 6.7 (cont.): Extracting key inferential results ===

print("\u2550\u2550\u2550 Significant Predictors (p < 0.05) \u2550\u2550\u2550\n")
sig = ols_result.pvalues[ols_result.pvalues < 0.05].sort_values()
for name, pval in sig.items():
    coef = ols_result.params[name]
    ci = ols_result.conf_int().loc[name]
    print(f"  {name:20s}  \u03b2 = {coef:+8.4f}  "
          f"p = {pval:.4f}  CI = [{ci[0]:+.4f}, {ci[1]:+.4f}]")

print(f"\n  Adj. R\u00b2      = {ols_result.rsquared_adj:.4f}")
print(f"  F-statistic  = {ols_result.fvalue:.2f}  "
      f"(p = {ols_result.f_pvalue:.2e})")
print("  Durbin\u2013Watson  = see summary table above")
print(f"\n\u2192 sklearn gives prediction; statsmodels gives inference.")
═══ Significant Predictors (p < 0.05) ═══

  model_year            β =  +0.7770  p = 0.0000  CI = [+0.6752, +0.8788]
  weight                β =  -0.0067  p = 0.0000  CI = [-0.0080, -0.0054]
  origin_1              β =  -5.8595  p = 0.0000  CI = [-8.2720, -3.4469]
  Intercept             β = -12.0951  p = 0.0006  CI = [-18.9406, -5.2497]
  displacement          β =  +0.0240  p = 0.0019  CI = [+0.0089, +0.0390]
  origin_2              β =  -3.2295  p = 0.0055  CI = [-5.5023, -0.9566]
  origin_3              β =  -3.0062  p = 0.0150  CI = [-5.4258, -0.5867]

  Adj. R²      = 0.8205
  F-statistic  = 224.45  (p = 1.79e-139)
  Durbin–Watson  = see summary table above

→ sklearn gives prediction; statsmodels gives inference.

Section 6.8 — Case Study: Life Expectancy (WHO)

We now apply every technique from this chapter to a real-world dataset published by the World Health Organisation — 2938 observations across 193 countries (2000–2015) with 22 attributes covering mortality, immunisation, economic, and social factors.

Target: Life expectancy (years)

This case study follows the complete modelling workflow:

  1. Exploratory Data Analysis — distributions, correlations, outliers
  2. Data Preparation — missing values, encoding, train/test split
  3. Model Fitting — OLS via sklearn, evaluate metrics
  4. Diagnostics — residual analysis
  5. Inferential Analysis — statsmodels summary
  6. Recommendations — which factors most influence life expectancy?
In [15]:
# === Section 6.8: Load WHO Life Expectancy dataset ===

df_life = pd.read_csv(DATA / "Life Expectancy Data.csv")
df_life.columns = df_life.columns.str.strip()
print(f"Shape: {df_life.shape}")
print(f"\nMissing values (top 10):")
print(df_life.isnull().sum().sort_values(ascending=False).head(10))
df_life.head()
Shape: (2938, 22)

Missing values (top 10):
Population                         652
Hepatitis B                        553
GDP                                448
Total expenditure                  226
Alcohol                            194
Income composition of resources    167
Schooling                          163
Thinness  1-19 years                34
Thinness 5-9 years                  34
BMI                                 34
dtype: int64
Out[15]:
Country Year Status Life expectancy Adult Mortality Infant deaths Alcohol Percentage expenditure Hepatitis B Measles ... Polio Total expenditure Diphtheria HIV/AIDS GDP Population Thinness 1-19 years Thinness 5-9 years Income composition of resources Schooling
0 Afghanistan 2015 Developing 65.0 263.0 62 0.01 71.279624 65.0 1154 ... 6.0 8.16 65.0 0.1 584.259210 33736494.0 17.2 17.3 0.479 10.1
1 Afghanistan 2014 Developing 59.9 271.0 64 0.01 73.523582 62.0 492 ... 58.0 8.18 62.0 0.1 612.696514 327582.0 17.5 17.5 0.476 10.0
2 Afghanistan 2013 Developing 59.9 268.0 66 0.01 73.219243 64.0 430 ... 62.0 8.13 64.0 0.1 631.744976 31731688.0 17.7 17.7 0.470 9.9
3 Afghanistan 2012 Developing 59.5 272.0 69 0.01 78.184215 67.0 2787 ... 67.0 8.52 67.0 0.1 669.959000 3696958.0 17.9 18.0 0.463 9.8
4 Afghanistan 2011 Developing 59.2 275.0 71 0.01 7.097109 68.0 3013 ... 68.0 7.87 68.0 0.1 63.537231 2978599.0 18.2 18.2 0.454 9.5

5 rows × 22 columns

In [16]:
# === Section 6.8: EDA — target distribution & correlation heatmap ===

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Target distribution
axes[0].hist(df_life["Life expectancy"].dropna(), bins=40, alpha=0.7,
             color="steelblue", edgecolor="white")
axes[0].axvline(df_life["Life expectancy"].mean(), color="red", ls="--",
                lw=1.5, label=f"Mean = {df_life['Life expectancy'].mean():.1f}")
axes[0].set_xlabel("Life Expectancy (years)")
axes[0].set_ylabel("Count")
axes[0].set_title("Distribution of Life Expectancy")
axes[0].legend()

# Correlation heatmap (numeric columns only)
numeric_cols = df_life.select_dtypes(include="number")
corr = numeric_cols.corr()
mask = np.triu(np.ones_like(corr, dtype=bool))
sns.heatmap(corr, mask=mask, cmap="RdBu_r", center=0, ax=axes[1],
            fmt=".1f", linewidths=0.5, square=True,
            cbar_kws={"shrink": 0.8})
axes[1].set_title("Feature Correlation Matrix")

fig.suptitle("WHO Life Expectancy — Exploratory Analysis", fontsize=14,
             y=1.02)
plt.tight_layout()
plt.show()

# Top correlations with target
target_corr = corr["Life expectancy"].drop("Life expectancy").sort_values()
print("═══ Top Correlations with Life Expectancy ═══")
print(f"\nStrongest positive:  {target_corr.iloc[-1]:.3f}  "
      f"({target_corr.index[-1]})")
print(f"Strongest negative:  {target_corr.iloc[0]:.3f}  "
      f"({target_corr.index[0]})")
No description has been provided for this image
═══ Top Correlations with Life Expectancy ═══

Strongest positive:  0.752  (Schooling)
Strongest negative:  -0.696  (Adult Mortality)
In [17]:
# === Section 6.8: EDA — scatter plots of top predictors ===

top_pos = ["Schooling", "Income composition of resources", "BMI"]
top_neg = ["Adult Mortality", "HIV/AIDS", "Infant deaths"]

fig, axes = plt.subplots(2, 3, figsize=(15, 8))

for ax, col in zip(axes[0], top_pos):
    ax.scatter(df_life[col], df_life["Life expectancy"], s=8, alpha=0.3,
               color="steelblue")
    ax.set_xlabel(col.strip())
    ax.set_ylabel("Life Expectancy")
    ax.set_title(f"{col.strip()} (positive)")

for ax, col in zip(axes[1], top_neg):
    ax.scatter(df_life[col], df_life["Life expectancy"], s=8, alpha=0.3,
               color="#ef8a62")
    ax.set_xlabel(col.strip())
    ax.set_ylabel("Life Expectancy")
    ax.set_title(f"{col.strip()} (negative)")

fig.suptitle("Key Predictors of Life Expectancy", fontsize=14, y=1.01)
plt.tight_layout()
plt.show()
No description has been provided for this image
In [18]:
# === Section 6.8: Data preparation — clean, encode, split ===

df_le = df_life.copy()

# Drop Country (high-cardinality, not a predictor for a general model)
df_le.drop(columns=["Country"], inplace=True)

# Encode 'Status' (Developing / Developed)
df_le = pd.get_dummies(df_le, columns=["Status"], dtype=int)

# Drop rows with missing target
df_le.dropna(subset=["Life expectancy"], inplace=True)

# Fill remaining NaN with column medians
df_le.fillna(df_le.median(numeric_only=True), inplace=True)

# Features & target
target_le = "Life expectancy"
features_le = [c for c in df_le.columns if c != target_le]

X_le = df_le[features_le].values
y_le = df_le[target_le].values

X_le_train, X_le_test, y_le_train, y_le_test = train_test_split(
    X_le, y_le, test_size=0.30, random_state=42
)

print(f"Training set: {X_le_train.shape}")
print(f"Test set:     {X_le_test.shape}")
print(f"Features:     {len(features_le)}")
Training set: (2049, 21)
Test set:     (879, 21)
Features:     21
In [19]:
# === Section 6.8: Model fitting & evaluation ===

model_le = LinearRegression().fit(X_le_train, y_le_train)

y_le_pred_train = model_le.predict(X_le_train)
y_le_pred_test  = model_le.predict(X_le_test)

r2_train  = r2_score(y_le_train, y_le_pred_train)
r2_test   = r2_score(y_le_test,  y_le_pred_test)
rmse_test = np.sqrt(mean_squared_error(y_le_test, y_le_pred_test))
mae_test  = mean_absolute_error(y_le_test, y_le_pred_test)

n_te, p_te = X_le_test.shape
r2_adj = 1 - (1 - r2_test) * (n_te - 1) / (n_te - p_te - 1)

print("═══ WHO Life Expectancy — Model Performance ═══\n")
print(f"  Training R²  = {r2_train:.4f}")
print(f"  Test R²      = {r2_test:.4f}")
print(f"  Adjusted R²  = {r2_adj:.4f}")
print(f"  RMSE         = {rmse_test:.2f} years")
print(f"  MAE          = {mae_test:.2f} years")
═══ WHO Life Expectancy — Model Performance ═══

  Training R²  = 0.5733
  Test R²      = 0.5851
  Adjusted R²  = 0.5750
  RMSE         = 6.22 years
  MAE          = 4.36 years
In [20]:
# === Section 6.8: Predicted vs. actual — Life Expectancy ===

fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(y_le_test, y_le_pred_test, alpha=0.4, s=20, color="steelblue",
           edgecolors="white", linewidths=0.3)
lims = [35, 92]
ax.plot(lims, lims, "r--", lw=1.5, label="Perfect prediction")
ax.set_xlim(lims)
ax.set_ylim(lims)
ax.set_xlabel("Actual Life Expectancy (years)")
ax.set_ylabel("Predicted Life Expectancy (years)")
ax.set_title("Predicted vs. Actual — WHO Life Expectancy")
ax.legend()
ax.set_aspect("equal")
plt.tight_layout()
plt.show()
No description has been provided for this image
In [21]:
# === Section 6.8: Residual diagnostics — Life Expectancy ===

resid_le = y_le_test - y_le_pred_test

fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))

axes[0].scatter(y_le_pred_test, resid_le, alpha=0.4, s=15,
                color="steelblue", edgecolors="white", linewidths=0.3)
axes[0].axhline(0, color="red", ls="--", lw=1.2)
axes[0].set_xlabel("Predicted")
axes[0].set_ylabel("Residual")
axes[0].set_title("Residuals vs. Predicted")

axes[1].hist(resid_le, bins=30, density=True, alpha=0.6,
             color="steelblue", edgecolor="white")
x_n = np.linspace(resid_le.min(), resid_le.max(), 200)
axes[1].plot(x_n, stats.norm.pdf(x_n, resid_le.mean(), resid_le.std()),
             "r-", lw=2)
axes[1].set_xlabel("Residual")
axes[1].set_ylabel("Density")
axes[1].set_title("Residual Distribution")

stats.probplot(resid_le, dist="norm", plot=axes[2])
axes[2].set_title("Normal Q–Q Plot")
axes[2].get_lines()[0].set_color("steelblue")
axes[2].get_lines()[0].set_alpha(0.6)

fig.suptitle("Residual Diagnostics — WHO Life Expectancy", fontsize=13,
             y=1.02)
plt.tight_layout()
plt.show()

stat_sw, p_sw = shapiro(resid_le[:500])  # Shapiro limited to ~5000
print(f"Shapiro–Wilk (first 500 residuals): W = {stat_sw:.4f}, p = {p_sw:.4f}")
No description has been provided for this image
Shapiro–Wilk (first 500 residuals): W = 0.8852, p = 0.0000
In [22]:
# === Section 6.8: statsmodels OLS — Life Expectancy ===

# Clean column names for the formula API
df_sm_le = df_le.copy()
df_sm_le.columns = (df_sm_le.columns
                    .str.strip()
                    .str.replace(" ", "_")
                    .str.replace("/", "_")
                    .str.replace("-", "_"))

target_clean = [c for c in df_sm_le.columns if "life_expectancy" in c.lower()][0]
feat_clean = [c for c in df_sm_le.columns if c != target_clean]

formula_le = f"{target_clean} ~ " + " + ".join(feat_clean)
ols_le = smf.ols(formula_le, data=df_sm_le).fit()
print(ols_le.summary())
                            OLS Regression Results                            
==============================================================================
Dep. Variable:        Life_expectancy   R-squared:                       0.820
Model:                            OLS   Adj. R-squared:                  0.819
Method:                 Least Squares   F-statistic:                     664.3
Date:                Sat, 08 Aug 2026   Prob (F-statistic):               0.00
Time:                        12:36:36   Log-Likelihood:                -8239.0
No. Observations:                2928   AIC:                         1.652e+04
Df Residuals:                    2907   BIC:                         1.665e+04
Df Model:                          20                                         
Covariance Type:            nonrobust                                         
===================================================================================================
                                      coef    std err          t      P>|t|      [0.025      0.975]
---------------------------------------------------------------------------------------------------
Intercept                          53.9440     23.270      2.318      0.021       8.317      99.570
Year                               -0.0126      0.017     -0.720      0.472      -0.047       0.022
Adult_Mortality                    -0.0199      0.001    -24.960      0.000      -0.021      -0.018
Infant_deaths                       0.0992      0.008     11.758      0.000       0.083       0.116
Alcohol                             0.0591      0.026      2.250      0.025       0.008       0.111
Percentage_expenditure           3.894e-05   9.07e-05      0.429      0.668      -0.000       0.000
Hepatitis_B                        -0.0160      0.004     -4.289      0.000      -0.023      -0.009
Measles                         -1.988e-05   7.66e-06     -2.595      0.010   -3.49e-05   -4.86e-06
BMI                                 0.0440      0.005      8.887      0.000       0.034       0.054
Under_five_deaths                  -0.0743      0.006    -12.010      0.000      -0.086      -0.062
Polio                               0.0288      0.004      6.470      0.000       0.020       0.038
Total_expenditure                   0.0569      0.035      1.639      0.101      -0.011       0.125
Diphtheria                          0.0413      0.005      8.889      0.000       0.032       0.050
HIV_AIDS                           -0.4709      0.018    -26.659      0.000      -0.506      -0.436
GDP                              4.236e-05   1.38e-05      3.066      0.002    1.53e-05    6.94e-05
Population                       8.199e-11   1.69e-09      0.049      0.961   -3.23e-09    3.39e-09
Thinness__1_19_years               -0.0815      0.050     -1.619      0.106      -0.180       0.017
Thinness_5_9_years                  0.0091      0.050      0.184      0.854      -0.088       0.106
Income_composition_of_resources     5.3361      0.642      8.312      0.000       4.077       6.595
Schooling                           0.6844      0.043     16.079      0.000       0.601       0.768
Status_Developed                   27.7719     11.628      2.388      0.017       4.971      50.572
Status_Developing                  26.1721     11.643      2.248      0.025       3.343      49.001
==============================================================================
Omnibus:                      136.024   Durbin-Watson:                   0.698
Prob(Omnibus):                  0.000   Jarque-Bera (JB):              387.766
Skew:                          -0.191   Prob(JB):                     6.27e-85
Kurtosis:                       4.741   Cond. No.                     5.88e+22
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The smallest eigenvalue is 2.56e-27. This might indicate that there are
strong multicollinearity problems or that the design matrix is singular.
In [23]:
# === Section 6.8: Key findings & recommendations ===

print("═══ Key Findings — WHO Life Expectancy ═══\n")

# Top 5 most significant positive & negative predictors
params = ols_le.params.drop("Intercept")
pvals = ols_le.pvalues.drop("Intercept")
sig_mask = pvals < 0.05
sig_params = params[sig_mask].sort_values()

print("Most significant NEGATIVE predictors (lower LE):")
for name, coef in sig_params.head(3).items():
    print(f"  → {name}: β = {coef:+.4f}  (p = {pvals[name]:.4f})")

print("\nMost significant POSITIVE predictors (higher LE):")
for name, coef in sig_params.tail(3).items():
    print(f"  → {name}: β = {coef:+.4f}  (p = {pvals[name]:.4f})")

print("\n═══ Recommendations ═══")
print("→ HIV/AIDS prevalence and adult mortality are the strongest")
print("  negative drivers — public health interventions in these areas")
print("  have the greatest potential to improve life expectancy.")
print("→ Schooling, income composition, and immunisation coverage are")
print("  the strongest positive drivers — investment in education and")
print("  healthcare access yields measurable improvements.")
print(f"\n  Overall model: Adj. R² = {ols_le.rsquared_adj:.4f}")
═══ Key Findings — WHO Life Expectancy ═══

Most significant NEGATIVE predictors (lower LE):
  → HIV_AIDS: β = -0.4709  (p = 0.0000)
  → Under_five_deaths: β = -0.0743  (p = 0.0000)
  → Adult_Mortality: β = -0.0199  (p = 0.0000)

Most significant POSITIVE predictors (higher LE):
  → Income_composition_of_resources: β = +5.3361  (p = 0.0000)
  → Status_Developing: β = +26.1721  (p = 0.0247)
  → Status_Developed: β = +27.7719  (p = 0.0170)

═══ Recommendations ═══
→ HIV/AIDS prevalence and adult mortality are the strongest
  negative drivers — public health interventions in these areas
  have the greatest potential to improve life expectancy.
→ Schooling, income composition, and immunisation coverage are
  the strongest positive drivers — investment in education and
  healthcare access yields measurable improvements.

  Overall model: Adj. R² = 0.8192

Section 6.9 — Summary & Quick Reference

Key Formulas

| Quantity | Formula | |:---------|:--------| | Simple regression | $\hat{y} = \beta_0 + \beta_1 x$ | | Multiple regression | $\hat{y} = \beta_0 + \sum_{j=1}^p \beta_j x_j$ | | OLS objective | $\min \sum(y_i - \hat{y}_i)^2$ | | $R^2$ | $1 - SSE/SST$ | | Adjusted $R^2$ | $1 - \dfrac{(1-R^2)(n-1)}{n-p-1}$ | | RMSE | $\sqrt{\dfrac{1}{n}\sum(y_i-\hat{y}_i)^2}$ | | MAE | $\dfrac{1}{n}\sum|y_i-\hat{y}_i|$ | | Ridge penalty | $+ \alpha \sum \beta_j^2$ | | Lasso penalty | $+ \alpha \sum |\beta_j|$ | | Z-score | $z = (x - \mu) / \sigma$ |

Python Cheat Sheet

from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures, scale
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error
import statsmodels.formula.api as smf

# OLS (sklearn):    model = LinearRegression().fit(X_train, y_train)
# Ridge:            model = Ridge(alpha=1.0).fit(X_train, y_train)
# Lasso:            model = Lasso(alpha=0.5).fit(X_train, y_train)
# Polynomial:       X_poly = PolynomialFeatures(2).fit_transform(X)
# Metrics:          r2_score(y, ŷ)  |  mean_squared_error(y, ŷ)
# statsmodels:      result = smf.ols("y ~ x1 + x2", data=df).fit()
# Summary:          result.summary()
# Significant:      result.pvalues[result.pvalues < 0.05]

Key Takeaways

$$\boxed{R^2 \text{ measures fit, Adjusted } R^2 \text{ penalises complexity, RMSE penalises large errors.}}$$

  • OLS is the foundation — understand it before moving to regularised methods.
  • Residual analysis is not optional: it reveals violated assumptions that invalidate inferences.
  • Ridge and Lasso are essential when you have many features or collinear predictors.
  • statsmodels complements sklearn by providing hypothesis tests on individual coefficients — connecting this chapter back to Chapter 5.
  • The case study showed that real-world regression involves extensive EDA, data cleaning, and domain-informed interpretation — not just calling .fit().

Chapter 2 gave you correlation; Chapter 5 gave you hypothesis tests on parameters; this chapter combined them into a complete predictive framework.


Next → Chapter 7: Classification & Model Selection


References

  1. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.). Springer. Chapter 3 covers linear regression at the level used here. https://www.statlearning.com/

  2. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer. Chapter 3 provides the graduate-level treatment of least squares and regularization (ridge, lasso). https://hastie.su.domains/ElemStatLearn/

  3. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B, 58(1), 267-288. The original lasso paper referenced by the regularization section.

  4. scikit-learn Developers. scikit-learn linear_model documentation. https://scikit-learn.org/stable/modules/linear_model.html

  5. statsmodels Development Team. statsmodels OLS documentation (diagnostics, VIF, residual plots). https://www.statsmodels.org/stable/regression.html