Descriptive Statistics for AI/ML¶
Chapter 2: From Raw Data to Information¶
Why This Chapter Matters¶
Before a model can learn from data, a human being has to understand what the data is. That sentence is simple, but it is the hinge on which responsible AI/ML turns. A table of numbers is not yet evidence. A collection of labels is not yet knowledge. Descriptive statistics is the first discipline that turns raw observations into intelligible information.
The path is the same in medicine, science, and machine learning: observe, organize, summarize, visualize, question, and only then model. A patient registry, a gene-expression matrix, an image dataset, a stream of telemetry, and a set of model scores all begin as measurements. They become meaningful only when we can describe their center, spread, shape, missingness, outliers, and relationships.
Chapter 1 gave the reader the Python vocabulary needed to compute. This chapter gives the statistical vocabulary needed to reason about data before modeling it. The goal is not to memorize formulas in isolation. The goal is to learn the sequence of thought: what kind of data do I have, what summaries are valid, what picture should I draw, what can this number hide, and what decision would be irresponsible before looking again?
The chapter proceeds from data provenance and measurement scales into measures of center, spread, shape, quantiles, visual summaries, and association. The computational examples use NumPy, Pandas, Matplotlib, Seaborn, and SciPy. For deeper technical reference, keep nearby the NumPy statistics routines, Pandas DataFrame.describe, Matplotlib histogram documentation, Seaborn statistical visualization guide, and SciPy stats reference.
By the end of the chapter, descriptive statistics should feel less like a list of tools and more like a professional habit: never hand data to a model until you have first described it honestly.
Section 2.0: Setup, Imports, and Reproducibility¶
Every computational chapter starts by making the execution environment explicit. This is not boilerplate. It is how a notebook becomes a scientific record rather than a memory of something that once worked on one machine.
The setup cell imports the working stack for descriptive statistics. NumPy provides fast arrays and numerical routines. Pandas gives us labeled tables. Matplotlib and Seaborn make distributions and relationships visible. SciPy supplies statistical functions such as skewness, probability plots, correlations, and contingency-table tests.
The cell also creates a random-number generator with a fixed seed. Random examples are useful only when they are reproducible. If a figure, statistic, or simulated dataset changes every time the notebook runs, the lesson becomes harder to verify. A controlled generator lets the reader re-run the chapter and see the same data, the same plots, and the same conclusions.
Run the setup cell before the later code cells. The later examples assume these imports, plotting defaults, and random generator are already available.
# === Chapter 2: Setup ===
# These are the standard imports for descriptive statistics work.
# NumPy: numerical arrays and math functions
# Pandas: DataFrames for tabular data
# Matplotlib + Seaborn: visualization
# SciPy.stats: statistical functions (skewness, kurtosis, etc.)
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import scipy
from scipy import stats
# Configure plots for clean rendering inside the notebook
plt.rcParams.update({
"figure.figsize": (8, 4), # default figure size (width, height) in inches
"axes.grid": True, # show gridlines by default
"grid.alpha": 0.3, # light gridlines (not distracting)
"axes.spines.top": False, # remove top border
"axes.spines.right": False, # remove right border
})
sns.set_theme(style="whitegrid")
# Seed for reproducibility -- a Generator instance gives the same results every run
rng = np.random.default_rng(42)
print("Chapter 2 environment ready.")
print(f"NumPy {np.__version__} | Pandas {pd.__version__} | SciPy {scipy.__version__}")
Section 2.1: Why Statistics and Big Data Belong Together¶
Statistical literacy is foundational for AI/ML because machine learning is not magic layered on top of data. It is estimation from data. A classifier learns parameters from examples. A clustering algorithm imposes structure on observed points. A neural network adjusts weights in response to measured error. In every case, the behavior of the model depends on the distribution, quality, and representativeness of the data.
The modern phrase big data can make this sound new, but the underlying problem is old. The scale has changed: electronic health records, imaging archives, genomic assays, sensors, mobile devices, web systems, and cloud platforms now produce data at a speed and volume that earlier statisticians could not have imagined. Yet the first questions remain statistical. How many observations are there? What is missing? Which variables are categorical, ordinal, or continuous? Are the tails heavy? Are the groups comparable? Are the labels trustworthy?
The common four-part description of big data is still useful if read carefully. Volume refers to the amount of data. Velocity refers to the speed at which it arrives. Variety refers to the many forms data can take: tables, text, images, audio, molecular measurements, and time series. Veracity refers to truthfulness and quality: measurement error, missingness, selection bias, label noise, and shifting definitions.
For the AI/ML practitioner, these are not abstract categories. They determine whether a dataset can support the question being asked. A model trained on a biased sample may perform well in validation and fail in deployment. A feature with extreme skew may dominate distance-based methods. A label collected under changing clinical definitions may encode practice patterns rather than biology.
Descriptive statistics is therefore the first defense against premature modeling. It makes the invisible structure of a dataset visible enough to question.
Section 2.2: What Statistics Does¶
Statistics is the discipline of collecting, organizing, summarizing, analyzing, and interpreting data under uncertainty. Historically, the word was connected to the state because early statistical work counted populations, taxes, resources, and military capacity. Today the same discipline underlies clinical trials, epidemiology, laboratory science, finance, manufacturing, social science, and machine learning.
The first major distinction is between descriptive and inferential statistics. Descriptive statistics summarizes the data we have. It includes tables, graphs, means, medians, variances, percentiles, correlations, and other summaries of observed values. Inferential statistics uses a sample to reason about a larger population or data-generating process. It includes confidence intervals, hypothesis tests, regression inference, Bayesian estimation, and related methods.
This chapter is about description, not because inference is less important, but because inference without description is fragile. Before asking what a population might be like, we must know what the observed sample looks like. Before asking whether two groups differ, we must know their distributions, sample sizes, outliers, and measurement scales. Before training a model, we must know what the model is being asked to learn from.
Descriptive statistics can also mislead. A mean can hide a bimodal distribution. A clean percentage can hide a small denominator. A correlation can be dominated by one outlier. A bar chart can exaggerate a trivial difference. This is why the chapter repeatedly pairs numerical summaries with visual inspection. One number is rarely enough.
Section 2.3: Population, Sample, Parameter, and Statistic¶
Four terms carry much of statistical reasoning: population, sample, parameter, and statistic.
The population is the complete set of observations we ultimately care about. In medical research, it might be all patients with a disease under a specified definition. In machine learning, it might be all future examples the model will encounter. The population is often conceptual rather than fully observable.
A parameter is a fixed but usually unknown property of that population, such as the true population mean, variance, or prevalence. We rarely know parameters directly because we rarely observe the full population.
A sample is the finite set of observations we actually have. A statistic is a number computed from that sample, such as a sample mean, sample variance, sample proportion, or test-set accuracy. Statistics estimate parameters, but the quality of that estimate depends on how the sample was obtained.
This distinction appears again in model evaluation. A test-set accuracy is a statistic computed from a particular held-out sample. The quantity we actually care about is the model's future performance on the population of unseen cases. If the test set is biased, too small, or contaminated by leakage, the statistic may be precise-looking and still misleading.
Sampling is therefore not an administrative detail. It is part of truth. Random sampling, stratified sampling, train-validation-test splitting, cross-validation, and external validation all exist because the path from sample to population is where many errors enter.
Section 2.4: Data Sources and Provenance¶
Data has a history. It was collected by someone, for some purpose, under some constraints, using some instrument, form, workflow, or platform. That history is called provenance, and it matters as much as the numerical values themselves.
Primary data is collected for the purpose at hand: a trial designed around a specific endpoint, a curated imaging dataset, a controlled experiment, or a purpose-built survey. It often gives the analyst better control over definitions and quality, but it is expensive and slow.
Secondary data was collected for another purpose and later reused. Examples include public repositories, billing records, government datasets, literature-derived corpora, and historical institutional data. Secondary data can be powerful, but its original definitions may not match the new question. Missing fields may reflect workflow rather than biology. Labels may reflect billing, convenience, or old clinical practice.
Current data arrives close to the present moment: streaming telemetry, live transactions, active monitoring data, or recent clinical operations. Archived data is historical. Both can be useful, but mixing them without thought introduces concept drift: the statistical relationship between features and outcomes may change over time.
For AI/ML, data provenance is not optional documentation. It is part of model validity. A model trained from one hospital, one device type, one demographic group, one time period, or one labeling convention may not generalize beyond that context. The data source tells us where the model's knowledge begins and where it may end.
Section 2.5: Types of Data¶
The first practical question in a dataset is not which model to use. It is what kind of values each column contains. Statistical summaries are meaningful only when they match the data type.
Qualitative data, also called categorical data, records labels or classes: blood type, tumor histology, treatment arm, hospital site, or mutation category. Some categorical variables have no natural order. Others are ordinal, meaning their categories have a rank order, such as pain scores, tumor stage, or response grades.
Quantitative data records numeric amount. Discrete quantitative variables are countable integers, such as number of visits, number of mutations, or number of positive nodes. Continuous quantitative variables can take values along an interval, such as age, blood pressure, weight, expression level, or probability score.
The distinction controls both computation and modeling. A mean is meaningful for a continuous laboratory value, questionable for an ordinal stage code, and meaningless for a nominal label. A categorical feature may need one-hot encoding, target encoding, or embeddings before a numerical model can use it. A continuous feature may need scaling, transformation, winsorization, or missing-value handling.
The next code cell builds a small Pandas table with categorical and quantitative columns so the distinction is visible in actual Python objects rather than only in prose.
# === Section 2.5: Types of Data in Python ===
# Let's create a small DataFrame to illustrate qualitative vs. quantitative data.
patients = pd.DataFrame({
# Qualitative (categorical) columns
"Name": ["Alice", "Bob", "Carol", "David", "Eve"],
"Gender": ["F", "M", "F", "M", "F"],
"Blood_Type": ["A+", "O-", "B+", "AB+", "O+"],
# Quantitative — Discrete
"Visits": [3, 1, 7, 2, 5],
# Quantitative — Continuous
"Temperature": [98.6, 99.1, 97.8, 100.4, 98.2],
"Weight_kg": [62.3, 85.1, 54.7, 91.0, 68.5],
})
print("Patient data (mixed types):")
print(patients)
print("\nPandas dtypes (note: object = string/categorical):")
print(patients.dtypes)
Section 2.6: Data Objects and Data Sets¶
A data object is the unit of observation. In a clinical table, the object may be a patient, encounter, specimen, lesion, claim, or medication order. In an image dataset, the object may be an image. In a natural-language corpus, it may be a document, paragraph, or token window. In Pandas, a data object is often represented as one row.
Attributes are the measured properties of those objects. If each patient is a row, then age, diagnosis, laboratory values, stage, treatment, and outcome are attributes. In machine learning language, the row becomes a feature vector: one point in a multidimensional feature space.
A data set is a collection of such objects. Most of this chapter uses tabular data because tables are the natural home of descriptive statistics and remain central in clinical, operational, and business AI. Other data sets have different structure: time series, images, graphs, text corpora, transaction baskets, or spatial coordinates. Each structure changes which summaries and models are appropriate.
The professional habit is to identify the object before summarizing the attributes. Confusion at this level causes subtle mistakes. A row might represent a patient, but it might also represent a visit, a lab result, or a billing line. Those are not interchangeable units of analysis.
Section 2.7: Data Attributes and Measurement Scales¶
An attribute is a measured characteristic of a data object. Different fields call attributes by different names: variables, features, predictors, dimensions, columns, or covariates. The name matters less than the measurement scale, because the scale determines which operations are valid.
Nominal attributes are labels with no intrinsic order. Blood type and tumor histology are examples. Binary attributes have two states; sometimes the two states are symmetric, and sometimes one state is the clinically important or rare event. Ordinal attributes have a meaningful rank order but not necessarily equal spacing between ranks. Interval attributes have ordered values and equal units but no true zero, such as Celsius temperature. Ratio attributes have equal units and a true zero, such as weight, age, concentration, count, or Kelvin temperature.
This classification prevents common errors. The average zip code is meaningless. Treating tumor stage as if the distance from I to II equals the distance from III to IV can mislead a model. Coding a rare positive diagnosis as a binary attribute may require special metrics because the positive state carries different information than the negative state.
Pandas supports categorical data directly through Categorical. Ordered categoricals allow the code to preserve rank information without pretending the values are ordinary continuous measurements.
# === Section 2.7: Attribute Types in Pandas ===
# Pandas has a 'Categorical' dtype that respects ordering.
# This matters for ordinal data.
# --- Nominal: no inherent order ---
colors = pd.Categorical(["red", "blue", "green", "red", "blue"])
print("Nominal (unordered):")
print(f" Categories: {colors.categories.tolist()}")
print(f" Ordered: {colors.ordered}\n")
# --- Ordinal: meaningful order ---
sizes = pd.Categorical(
["Medium", "Large", "Small", "Large", "Medium"],
categories=["Small", "Medium", "Large"], # define the order
ordered=True # mark as ordinal
)
print("Ordinal (ordered):")
print(f" Categories: {sizes.categories.tolist()}")
print(f" Ordered: {sizes.ordered}")
print(f" 'Medium' < 'Large'? {sizes[0] < sizes[1]}\n") # comparison works!
# --- Binary (asymmetric) ---
diagnoses = pd.Series([1, 0, 0, 1, 0], name="Diagnosis")
print("Binary (asymmetric — 1 = positive):")
print(f" Positive rate: {diagnoses.mean():.0%}")
Section 2.8: The Descriptive Statistics Toolkit¶
Descriptive statistics is a toolkit, not one number. A dataset has center, spread, shape, boundaries, outliers, missingness, and relationships among variables. Different tools reveal different parts of that structure.
Measures of central tendency ask where the data tends to sit: mean, median, and mode. Measures of dispersion ask how widely the values vary: range, interquartile range, variance, standard deviation, and coefficient of variation. Shape summaries such as skewness and kurtosis ask whether the distribution is symmetric, lopsided, light-tailed, or heavy-tailed. Quantile methods ask where observations fall by rank. Visual displays such as box plots, histograms, quantile plots, Q-Q plots, scatter plots, and heatmaps show patterns that numbers can hide. Association measures such as Pearson correlation and chi-square statistics ask how variables move together.
The order matters. First identify the data types and measurement scales. Then compute summaries that are valid for those scales. Then visualize the distributions. Then examine outliers and relationships. This is the descriptive workflow that should precede any serious modeling step.
For practical reference, Pandas describe, NumPy's statistics routines, and SciPy's stats documentation are the main technical manuals behind the code that follows.
Section 2.9: Central Tendency¶
The first question about a numeric variable is often: where is its center? Measures of central tendency answer that question in different ways.
The arithmetic mean adds all values and divides by the number of observations. It uses every point, which makes it mathematically convenient and statistically powerful under the right conditions. It is also sensitive to extreme values. One erroneous or genuinely extreme observation can move the mean substantially.
The median is the middle value after sorting. It depends on rank rather than magnitude, so it is robust to outliers and skewed distributions. Income, length of stay, claims cost, survival time, and many biological measures are often better summarized by a median than by a mean.
The mode is the most frequent value. It is the only member of this trio that naturally applies to nominal categorical data. A tumor histology can have a most common category; it cannot have a meaningful average category.
Choosing among mean, median, and mode is not taste. It depends on distribution shape and measurement scale. Symmetric continuous data often support the mean. Skewed data or data with extreme values often call for the median. Categorical data calls for counts, proportions, and modes.
The following code computes all three measures and then shows how a single outlier can distort the mean while barely moving the median.
# === Section 2.9: Central Tendency ===
# We'll compute mean, median, and mode on a sample dataset.
# Sample data: exam scores for a class of 15 students
scores = np.array([72, 85, 90, 65, 78, 92, 88, 76, 95, 70, 85, 60, 85, 82, 91])
# --- Mean ---
mean_score = np.mean(scores) # or scores.mean()
# --- Median ---
median_score = np.median(scores) # middle value of sorted data
# --- Mode ---
# NumPy doesn't have a mode function; use scipy.stats
mode_result = stats.mode(scores, keepdims=True)
mode_score = mode_result.mode[0]
mode_count = mode_result.count[0]
print(f"Exam scores: {sorted(scores)}")
print(f"\n Mean: {mean_score:.2f} (sum / n = {scores.sum()} / {len(scores)})")
print(f" Median: {median_score:.1f} (middle value when sorted)")
print(f" Mode: {mode_score} (appears {mode_count} times)")
print(f"\n Mean > Median? {mean_score > median_score} → suggests slight left skew")
# === Effect of Outliers on Mean vs. Median ===
# This demonstrates WHY median is preferred for skewed data.
# Normal scores
normal = np.array([50, 55, 60, 62, 65, 68, 70, 72, 75, 80])
# Same scores but with one extreme outlier (a student scored 200 by error)
with_outlier = np.array([50, 55, 60, 62, 65, 68, 70, 72, 75, 200])
print("Without outlier:")
print(f" Mean = {normal.mean():.1f}, Median = {np.median(normal):.1f}")
print("\nWith outlier (200 replaces 80):")
print(f" Mean = {with_outlier.mean():.1f}, Median = {np.median(with_outlier):.1f}")
print("\n→ The mean jumped from 65.7 to 77.7 (+12), but the median barely changed.")
print(" This is why median is called a ROBUST measure of central tendency.")
Section 2.10: Skewness¶
Skewness describes asymmetry. A symmetric distribution has tails that balance each other. A right-skewed distribution has a longer right tail. A left-skewed distribution has a longer left tail. The name follows the tail, not the side where most observations sit.
Skewness is part of a larger moment hierarchy. The first moment describes location through the mean. The second describes spread through variance. The third describes asymmetry through skewness. The fourth, kurtosis, describes tail heaviness. Each layer adds information that the previous layer cannot show alone.
Skewness matters because many methods behave best when features are approximately symmetric. Strong skew can distort linear models, distance calculations, clustering, optimization, and visual interpretation. Common remedies include log transforms, Box-Cox transforms, quantile transforms, and robust summaries.
The point is not to force every variable to become normal. The point is to notice when shape matters. A long tail may be a nuisance, but it may also be the phenomenon of interest: rare high-cost claims, unusually long survival, extreme expression, or high-risk events.
# === Section 2.10: Skewness ===
# Generate three distributions: symmetric, right-skewed, left-skewed
symmetric = np.random.normal(loc=50, scale=10, size=1000) # normal = symmetric
right_skew = np.random.exponential(scale=10, size=1000) # exponential = right-skewed
left_skew = 100 - np.random.exponential(scale=10, size=1000) # mirror = left-skewed
fig, axes = plt.subplots(1, 3, figsize=(14, 4))
for ax, data, title in zip(
axes,
[left_skew, symmetric, right_skew],
["Left-Skewed (negative)", "Symmetric", "Right-Skewed (positive)"]
):
ax.hist(data, bins=30, edgecolor="white", alpha=0.7)
ax.axvline(np.mean(data), color="red", linestyle="--", label=f"Mean = {np.mean(data):.1f}")
ax.axvline(np.median(data), color="blue", linestyle="-", label=f"Median = {np.median(data):.1f}")
skew_val = stats.skew(data)
ax.set_title(f"{title}\nSkewness = {skew_val:.2f}")
ax.legend(fontsize=8)
plt.tight_layout()
plt.show()
Section 2.11: Measures of Dispersion¶
Center is not enough. Two variables can have the same mean and entirely different behavior. One may be tightly clustered; the other may swing widely. Dispersion describes how far observations spread around the center.
The range is the simplest measure: maximum minus minimum. It is easy to compute and easy to understand, but it uses only two observations. A single data-entry error, instrument artifact, or true extreme value can dominate it.
More useful measures look deeper into the distribution. The interquartile range focuses on the middle half of the data and is resistant to outliers. Variance and standard deviation use every observation and become central to modeling, inference, optimization, and feature scaling. The coefficient of variation expresses spread relative to the mean, which helps compare variables with different units or magnitudes.
In machine learning, dispersion is practical. Features with large raw scales can dominate distance-based algorithms. Low-variance features may carry little signal. Highly variable features may need transformation, clipping, or robust scaling. Describing spread is therefore both statistical and computational preparation.
Section 2.12: The Five-Number Summary¶
The five-number summary gives a compact view of a distribution: minimum, first quartile, median, third quartile, and maximum. It is one of the most useful summaries in exploratory data analysis because it combines boundary, center, spread, and asymmetry.
The quartiles divide ordered data into quarters. The first quartile, Q1, is near the 25th percentile. The median, Q2, is the 50th percentile. The third quartile, Q3, is near the 75th percentile. The interquartile range is Q3 minus Q1, representing the middle half of the data.
The IQR is robust because it ignores the most extreme quarters of the distribution. A maximum can change dramatically without changing Q1, the median, or Q3. This is why quartile-based summaries are central to box plots and outlier screening.
Quartile definitions vary across software and textbooks. NumPy's percentile functions use documented interpolation methods. Hand calculations may use slightly different conventions. This is not a contradiction; it is a reminder to report the method when exact quartile values matter.
The next code cells compute the five-number summary using NumPy and then implement a hand-style quartile algorithm so the reader can see both the concept and the convention.
# === Section 2.12: Five-Number Summary ===
# Using the worked example from the manuscript
data = np.array([1, 2, 4, 6, 8, 9, 10, 11, 14, 15, 16, 18, 30])
# --- NumPy approach ---
minimum = data.min()
q1 = np.percentile(data, 25)
median = np.median(data)
q3 = np.percentile(data, 75)
maximum = data.max()
iqr = q3 - q1
print(f"Data (sorted): {data}")
print(f"n = {len(data)}\n")
print("Five-Number Summary:")
print(f" Min = {minimum}")
print(f" Q1 = {q1}")
print(f" Q2 = {median} (median)")
print(f" Q3 = {q3}")
print(f" Max = {maximum}")
print(f"\n IQR = Q3 - Q1 = {q3} - {q1} = {iqr}")
print(f" Range = Max - Min = {maximum} - {minimum} = {maximum - minimum}")
# === Quartile Algorithm from the Manuscript (translated to Python) ===
# The manuscript gives pseudocode for finding quartiles.
# Here it is as working Python.
import math
def find_quartiles(data_list):
"""Find Q1, Q2, Q3 using the manuscript's algorithm."""
sorted_data = sorted(data_list)
n = len(sorted_data)
quartiles = {}
for k, name in [(1, "Q1"), (2, "Q2"), (3, "Q3")]:
pos = n * k / 4
if pos == int(pos): # position is an integer
# Quartile = mean of values at positions pos and pos+1 (1-indexed)
idx = int(pos) - 1 # convert to 0-indexed
quartiles[name] = (sorted_data[idx] + sorted_data[idx + 1]) / 2
else:
# Round up — value at that position is the quartile
idx = math.ceil(pos) - 1 # convert to 0-indexed
quartiles[name] = sorted_data[idx]
return quartiles
# Test with the manuscript example
example = [1, 2, 4, 6, 8, 9, 10, 11, 14, 15, 16, 18, 30]
q = find_quartiles(example)
print(f"Data: {example}")
print(f"n = {len(example)}")
print(f"\nManuscript algorithm:")
for name, val in q.items():
print(f" {name} = {val}")
print(f" IQR = {q['Q3'] - q['Q1']}")
print(f"\nNumPy (linear interpolation):")
print(f" Q1 = {np.percentile(example, 25)}, Q2 = {np.percentile(example, 50)}, Q3 = {np.percentile(example, 75)}")
print("\n→ Slight differences are expected — different methods, same concept.")
Section 2.13: Variance¶
Variance measures average squared distance from the mean. It is the second central moment of a distribution and one of the central quantities in statistics, optimization, signal processing, finance, and machine learning.
For a full population, variance divides the sum of squared deviations by the population size. For a sample, the usual sample variance divides by n - 1. That denominator is Bessel's correction. It accounts for the fact that the sample mean was estimated from the same data and therefore one degree of freedom has already been used.
The key idea is that deviations from the sample mean must sum to zero. Once n - 1 deviations are known, the last one is forced. Dividing by n would systematically underestimate the population variance when working from samples. Dividing by n - 1 corrects that bias under the usual assumptions.
NumPy makes this choice explicit through ddof, the delta degrees of freedom. np.var(x, ddof=0) computes population variance. np.var(x, ddof=1) computes sample variance. The same convention appears in np.std.
Variance also leads directly to covariance. Variance describes one variable's scatter around its own mean. Covariance describes whether two variables deviate from their means together. Correlation, PCA, regression, and portfolio theory all build on this idea.
# === Section 2.13: Variance — Step by Step ===
# Let's compute variance manually and verify with NumPy.
data = np.array([4, 8, 6, 5, 3, 2, 8, 9, 2, 5])
n = len(data)
mean = data.mean()
# Step 1: deviations from the mean
deviations = data - mean
# Step 2: squared deviations
sq_deviations = deviations ** 2
# Step 3: sum of squared deviations
ss = sq_deviations.sum()
# Step 4: divide by (n - 1) for sample variance
sample_var = ss / (n - 1)
# Step 5: divide by n for population variance
pop_var = ss / n
print(f"Data: {data}")
print(f"Mean: {mean}")
print(f"Deviations: {deviations}")
print(f"Squared deviations: {sq_deviations}")
print(f"Sum of squares: {ss}")
print(f"\nSample variance (÷ n-1 = {n-1}): {sample_var:.4f}")
print(f"Population variance (÷ n = {n}): {pop_var:.4f}")
# Verify with NumPy
print(f"\nNumPy np.var(data, ddof=1): {np.var(data, ddof=1):.4f} (sample)")
print(f"NumPy np.var(data, ddof=0): {np.var(data, ddof=0):.4f} (population)")
print("\n→ ddof = 'delta degrees of freedom'. ddof=1 uses n-1 (sample), ddof=0 uses n (population).")
Section 2.14: Standard Deviation¶
Variance is powerful but awkward to interpret because it is measured in squared units. If the original variable is measured in centimeters, variance is measured in square centimeters. Standard deviation solves this by taking the square root of variance, returning spread to the original unit.
Standard deviation is the most widely reported measure of dispersion in scientific work. A result written as mean plus or minus standard deviation is telling the reader both center and typical spread. The interpretation is strongest when the distribution is approximately normal.
For a normal distribution, the empirical rule says that about 68 percent of observations fall within one standard deviation of the mean, about 95 percent within two, and about 99.7 percent within three. This rule is useful because it turns standard deviation into a mental ruler.
The rule is not universal. Skewed, heavy-tailed, multimodal, and bounded distributions can depart sharply from those percentages. When normality is uncertain, visualize the data and consider distribution-free tools such as Chebyshev's inequality.
The next code cell simulates normally distributed data and counts how many observations fall within one, two, and three standard deviations.
# === Section 2.14: Standard Deviation & the 68-95-99 Rule ===
# Generate normally distributed data (mean=100, stdev=15, n=10000)
# Think of this as IQ scores
data = np.random.normal(loc=100, scale=15, size=10_000)
mean = data.mean()
std = data.std(ddof=1) # sample standard deviation
# Count how many fall within 1, 2, 3 standard deviations
within_1 = np.sum((data >= mean - 1*std) & (data <= mean + 1*std)) / len(data) * 100
within_2 = np.sum((data >= mean - 2*std) & (data <= mean + 2*std)) / len(data) * 100
within_3 = np.sum((data >= mean - 3*std) & (data <= mean + 3*std)) / len(data) * 100
print(f"Data: {len(data):,} normally distributed values")
print(f"Mean = {mean:.2f}, StDev = {std:.2f}\n")
print(f"Within ±1σ ({mean-std:.0f} to {mean+std:.0f}): {within_1:.1f}% (expected ~68%)")
print(f"Within ±2σ ({mean-2*std:.0f} to {mean+2*std:.0f}): {within_2:.1f}% (expected ~95%)")
print(f"Within ±3σ ({mean-3*std:.0f} to {mean+3*std:.0f}): {within_3:.1f}% (expected ~99.7%)")
# --- Visualize ---
fig, ax = plt.subplots(figsize=(10, 4))
ax.hist(data, bins=60, density=True, alpha=0.6, edgecolor="white", color="steelblue")
# Overlay the theoretical normal curve
x = np.linspace(data.min(), data.max(), 200)
ax.plot(x, stats.norm.pdf(x, mean, std), "r-", linewidth=2, label="Normal curve")
# Shade the 1σ, 2σ, 3σ bands
colors = ["#2166ac", "#67a9cf", "#d1e5f0"]
labels = ["±1σ (68%)", "±2σ (95%)", "±3σ (99.7%)"]
for k, (c, lbl) in enumerate(zip(colors[::-1], labels[::-1]), 1):
k_rev = 4 - k # shade from outside in so colors layer correctly
lo, hi = mean - k_rev*std, mean + k_rev*std
x_fill = np.linspace(lo, hi, 200)
ax.fill_between(x_fill, stats.norm.pdf(x_fill, mean, std), alpha=0.3, color=c, label=lbl)
ax.set_title("The 68–95–99.7 Rule (Empirical Rule)", fontsize=13)
ax.set_xlabel("Value")
ax.set_ylabel("Density")
ax.legend(loc="upper right")
plt.tight_layout()
plt.show()
Section 2.15: Coefficient of Variation¶
Standard deviation is an absolute measure of spread. The coefficient of variation is a relative measure: standard deviation divided by the mean, usually expressed as a percentage.
Relative spread matters when variables have different units or very different magnitudes. A standard deviation of 10 is small for a variable with mean 1000 and large for a variable with mean 12. The coefficient of variation puts spread on a comparable scale by asking how large the standard deviation is relative to the mean.
The CV is useful in laboratory reproducibility, finance, manufacturing, and feature comparison. It can help compare assay variability, return variability, or feature variability before modeling. It is meaningful mainly for ratio-scale data with a true zero and a mean safely away from zero. If the mean is near zero, the ratio becomes unstable.
The code example compares relative variability and verifies the calculation with SciPy's stats.variation.
# === Section 2.15: Coefficient of Variation ===
# Compare two datasets with the same standard deviation but different means
heights = np.array([165, 170, 175, 168, 180, 162, 172, 178, 169, 171])
weights = np.array([60, 70, 75, 68, 80, 55, 72, 78, 65, 67])
for name, arr in [("Heights (cm)", heights), ("Weights (kg)", weights)]:
m = arr.mean()
s = arr.std(ddof=1)
cv = (s / m) * 100
print(f"{name:20s} Mean = {m:.1f}, StDev = {s:.1f}, CV = {cv:.1f}%")
print("\n→ Even though the raw standard deviations differ,")
print(" CV tells us which variable is more dispersed RELATIVE to its mean.")
# scipy has stats.variation() for this
print(f"\nscipy.stats.variation(heights): {stats.variation(heights, ddof=1):.4f}")
print(f"scipy.stats.variation(weights): {stats.variation(weights, ddof=1):.4f}")
Section 2.16: Chebyshev's Rule¶
The empirical rule is elegant, but it depends on approximate normality. Chebyshev's inequality gives a weaker but more general guarantee: for any distribution with finite variance, at least 1 - 1 / k^2 of observations lie within k standard deviations of the mean for k > 1.
The guarantee is conservative. For normal data, the true percentages within two or three standard deviations are much higher than Chebyshev requires. The value of the rule is that it works without assuming normality. It applies to skewed, heavy-tailed, and irregular distributions.
This matters in exploratory analysis because many real datasets are not normal. Claims costs, gene expression, utilization counts, web traffic, transaction amounts, and many clinical variables can be skewed or heavy-tailed. Chebyshev's rule gives a baseline statement about spread even before the distribution is modeled.
The code example uses exponential data to show that the actual fraction within each standard-deviation band exceeds the Chebyshev lower bound.
# === Section 2.16: Chebyshev's Rule ===
# Demonstrate that Chebyshev holds even for non-normal data.
# Generate highly skewed data (NOT normal)
skewed_data = np.random.exponential(scale=10, size=10_000)
mean = skewed_data.mean()
std = skewed_data.std(ddof=1)
print("Chebyshev's Rule applied to SKEWED (exponential) data")
print(f"Mean = {mean:.2f}, StDev = {std:.2f}\n")
print(f"{'k':>3s} {'Chebyshev bound':>16s} {'Actual %':>10s} {'Satisfies?':>10s}")
print("-" * 50)
for k in [1.5, 2, 2.5, 3, 4]:
chebyshev_pct = (1 - 1/k**2) * 100
actual_pct = np.mean((skewed_data >= mean - k*std) & (skewed_data <= mean + k*std)) * 100
ok = "✓" if actual_pct >= chebyshev_pct else "✗"
print(f"{k:3.1f} {chebyshev_pct:15.1f}% {actual_pct:9.1f}% {ok:>10s}")
print("\n→ The actual percentage ALWAYS exceeds Chebyshev's bound.")
print(" Chebyshev is a guaranteed lower limit for ANY distribution.")
Section 2.17: Box and Whisker Plots¶
A box plot is the visual form of the five-number summary. It shows the median, the interquartile range, whiskers, and potential outliers in a compact display. It is especially useful for comparing distributions across groups.
The box extends from Q1 to Q3, so its height or width represents the IQR. The line inside the box is the median. Whiskers extend to non-outlying values according to a rule, commonly Tukey's 1.5 times IQR fence. Points beyond the fences are plotted individually as potential outliers.
The word potential is important. A point beyond the whisker is not automatically wrong. It may be a measurement error, a rare but real case, a subgroup, a data-entry problem, or the most clinically important observation in the dataset. The plot tells us where to investigate.
Box plots are often more informative than bar charts for continuous data because they show spread and asymmetry rather than only a mean. Side-by-side box plots can immediately reveal group differences, outlier burden, and heterogeneity.
# === Section 2.17: Box and Whisker Plots ===
# Use the manuscript example + a few extra points to show outliers
data = np.array([1, 2, 4, 6, 8, 9, 10, 11, 14, 15, 16, 18, 30])
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
# --- Basic box plot ---
bp = axes[0].boxplot(data, vert=True, widths=0.5, patch_artist=True,
boxprops=dict(facecolor="#67a9cf", alpha=0.7),
medianprops=dict(color="red", linewidth=2))
axes[0].set_title("Box Plot — Manuscript Example")
axes[0].set_ylabel("Value")
# Annotate the five-number summary
q1, med, q3 = np.percentile(data, [25, 50, 75])
axes[0].annotate(f"Min = {data.min()}", xy=(1, data.min()), xytext=(1.3, data.min()),
fontsize=9, arrowprops=dict(arrowstyle="->"))
axes[0].annotate(f"Q1 = {q1}", xy=(1, q1), xytext=(1.3, q1),
fontsize=9, arrowprops=dict(arrowstyle="->"))
axes[0].annotate(f"Median = {med}", xy=(1, med), xytext=(1.3, med),
fontsize=9, arrowprops=dict(arrowstyle="->"), color="red")
axes[0].annotate(f"Q3 = {q3}", xy=(1, q3), xytext=(1.3, q3),
fontsize=9, arrowprops=dict(arrowstyle="->"))
axes[0].annotate(f"Max = {data.max()} (outlier)", xy=(1, data.max()), xytext=(1.3, data.max()),
fontsize=9, arrowprops=dict(arrowstyle="->"))
# --- Comparing distributions side by side ---
group_a = rng.normal(50, 10, 200) # narrow spread
group_b = rng.normal(50, 25, 200) # wide spread
group_c = np.concatenate([rng.normal(30, 5, 150),
rng.normal(70, 5, 50)]) # bimodal
bp2 = axes[1].boxplot([group_a, group_b, group_c], vert=True,
tick_labels=["Narrow\n(σ=10)", "Wide\n(σ=25)", "Bimodal"],
patch_artist=True,
medianprops=dict(color="red", linewidth=2))
colors = ["#67a9cf", "#ef8a62", "#b2df8a"]
for patch, color in zip(bp2["boxes"], colors):
patch.set_facecolor(color)
patch.set_alpha(0.7)
axes[1].set_title("Comparing Distributions with Box Plots")
axes[1].set_ylabel("Value")
plt.tight_layout()
plt.show()
Section 2.18: Histograms¶
A histogram shows the shape of a distribution by dividing the range into bins and counting how many observations fall into each bin. Where a box plot is compact, a histogram is more revealing. It can show symmetry, skewness, multimodality, gaps, boundaries, and unusual spikes.
The bin choice matters. Too few bins blur structure. Too many bins make noise look like pattern. Rules such as Sturges, Scott, and Freedman-Diaconis provide starting points, but the analyst should still look critically at the result.
Histograms can display raw frequency, relative frequency, cumulative frequency, or cumulative relative frequency. The cumulative relative view is often called an ogive. It connects histograms to percentiles: the y-axis tells what fraction of observations are at or below a value.
Modern analysis often overlays a kernel density estimate on a histogram. A KDE is a smoothed estimate of the underlying distribution. It is not a replacement for judgment, but it can help the eye see the distribution's broad shape.
# === Section 2.18: Histograms — Four Views of the Same Data ===
data = rng.normal(loc=70, scale=12, size=200) # exam scores
fig, axes = plt.subplots(2, 2, figsize=(12, 9))
# (a) Frequency histogram
counts, bin_edges, _ = axes[0, 0].hist(data, bins=15, edgecolor="white",
color="steelblue", alpha=0.7)
axes[0, 0].set_title("(a) Frequency Histogram")
axes[0, 0].set_ylabel("Count")
# (b) Relative frequency histogram
axes[0, 1].hist(data, bins=15, edgecolor="white", color="#ef8a62",
alpha=0.7, density=False, weights=np.ones(len(data)) / len(data))
axes[0, 1].set_title("(b) Relative Frequency Histogram")
axes[0, 1].set_ylabel("Proportion")
# (c) Cumulative frequency histogram
axes[1, 0].hist(data, bins=15, edgecolor="white", color="#b2df8a",
alpha=0.7, cumulative=True)
axes[1, 0].set_title("(c) Cumulative Frequency")
axes[1, 0].set_ylabel("Cumulative Count")
# (d) Ogive (Cumulative Relative Frequency)
sorted_data = np.sort(data)
crf = np.arange(1, len(sorted_data) + 1) / len(sorted_data)
axes[1, 1].plot(sorted_data, crf, color="purple", linewidth=2)
axes[1, 1].axhline(0.5, color="gray", linestyle="--", alpha=0.5, label="50th percentile")
axes[1, 1].set_title("(d) Ogive (Cumulative Relative Frequency)")
axes[1, 1].set_ylabel("Cumulative Proportion")
axes[1, 1].legend()
for ax in axes.flat:
ax.set_xlabel("Value")
plt.suptitle("Four Views of the Same Distribution", fontsize=14, y=1.01)
plt.tight_layout()
plt.show()
Section 2.19: Quantile Plots¶
A quantile plot displays ordered observations against their percentile positions. Unlike a histogram, it does not bin the data. Every observation remains visible through its rank.
The construction is straightforward. Sort the values. Assign each sorted value a plotting position such as (i - 0.5) / n. Plot value against percentile. The resulting curve shows how quickly values rise across the distribution.
Flat regions indicate many observations packed into a narrow value range. Steep regions indicate sparse data, where a small change in percentile corresponds to a large change in value. Gaps and jumps may reveal discreteness, measurement thresholds, or subgroups.
Quantile plots are useful because they train the reader to think in ranks. Many robust statistics, including the median, quartiles, IQR, and percentile-based thresholds, come from rank order rather than raw magnitude.
# === Section 2.19: Quantile Plot ===
# Using the manuscript example
y = np.array([1, 2, 4, 6, 8, 9, 10, 11, 14, 15])
n = len(y)
percentiles = [(i - 0.5) / n * 100 for i in range(1, n + 1)]
fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(percentiles, y, "o-", color="steelblue", markersize=8)
# Label each point
for p, val in zip(percentiles, y):
ax.annotate(f"{val}", (p, val), textcoords="offset points",
xytext=(8, 5), fontsize=9)
ax.set_xlabel("Percentile")
ax.set_ylabel("Data Value")
ax.set_title("Quantile Plot")
ax.set_xticks(range(0, 101, 10))
plt.tight_layout()
plt.show()
# Print the table
print("\nPercentile Table:")
print(f"{'Value':>6s} {'Percentile':>10s}")
for p, val in zip(percentiles, y):
print(f"{val:6.0f} {p:10.0f}th")
Section 2.20: Quantile-Quantile Plots¶
A quantile-quantile plot, usually called a Q-Q plot, compares two distributions by plotting their corresponding quantiles against each other. If the distributions have the same shape, the points fall close to a straight line. If they differ, the pattern of departure tells us how.
The most common Q-Q plot compares sample data with a theoretical normal distribution. If the points roughly follow the reference line, normality may be a reasonable approximation. Curvature suggests skewness. Tail departures suggest heavier or lighter tails than the normal distribution. Isolated points may indicate outliers.
Q-Q plots can also compare two empirical samples, such as treatment and control groups. If points fall below the diagonal, one sample tends to have smaller values at corresponding quantiles. If the pattern bends, the groups differ in more than just location.
The Q-Q plot is diagnostic, not decorative. It answers a question that a mean and standard deviation cannot answer: is the entire distribution behaving like the comparison distribution?
# === Section 2.20: Q-Q Plots ===
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
# (a) Normal data → Q-Q against normal should be a straight line
normal_data = rng.normal(loc=50, scale=10, size=200)
stats.probplot(normal_data, dist="norm", plot=axes[0])
axes[0].set_title("(a) Normal Data\n(points on the line)")
axes[0].get_lines()[0].set_markerfacecolor("steelblue")
axes[0].get_lines()[0].set_markersize(4)
# (b) Right-skewed data → departs at the tails
skewed_data = rng.exponential(scale=10, size=200)
stats.probplot(skewed_data, dist="norm", plot=axes[1])
axes[1].set_title("(b) Right-Skewed Data\n(curves up at right tail)")
axes[1].get_lines()[0].set_markerfacecolor("#ef8a62")
axes[1].get_lines()[0].set_markersize(4)
# (c) Heavy-tailed data (t-distribution with 3 df)
heavy_tail = rng.standard_t(df=3, size=200)
stats.probplot(heavy_tail, dist="norm", plot=axes[2])
axes[2].set_title("(c) Heavy-Tailed Data\n(departs at both tails)")
axes[2].get_lines()[0].set_markerfacecolor("#b2df8a")
axes[2].get_lines()[0].set_markersize(4)
plt.suptitle("Q-Q Plots: Comparing Samples Against a Normal Distribution",
fontsize=13, y=1.02)
plt.tight_layout()
plt.show()
print("Interpretation:")
print(" (a) Points closely follow the line → data IS normally distributed")
print(" (b) Points curve up at the right → right-skewed (longer right tail)")
print(" (c) Points depart at both ends → heavier tails than normal (more extreme values)")
# === Q-Q Plot: Comparing Two Samples Against Each Other ===
# Using the manuscript example with two datasets
x = np.array([2, 3, 5, 7, 9, 12, 14, 16, 18, 20])
y = np.array([1, 2, 4, 6, 8, 9, 10, 11, 14, 15])
# When both have the same number of points, just plot sorted x vs sorted y
x_sorted = np.sort(x)
y_sorted = np.sort(y)
fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(x_sorted, y_sorted, color="steelblue", s=60, zorder=3)
# Reference line (y = x)
lims = [min(x_sorted.min(), y_sorted.min()), max(x_sorted.max(), y_sorted.max())]
ax.plot(lims, lims, "r--", alpha=0.6, label="y = x (identical distributions)")
# Label selected points from the manuscript
for xi, yi in zip(x_sorted, y_sorted):
ax.annotate(f"({xi},{yi})", (xi, yi), textcoords="offset points",
xytext=(6, 6), fontsize=8)
ax.set_xlabel("x quantiles")
ax.set_ylabel("y quantiles")
ax.set_title("Q-Q Plot: Dataset x vs. Dataset y")
ax.legend()
ax.set_aspect("equal")
plt.tight_layout()
plt.show()
print("Points below the diagonal → y values are generally smaller than x values.")
print("If points were ON the diagonal, the two datasets would have identical distributions.")
Section 2.21: Scatter Plots¶
Up to this point, most tools have described one variable at a time. A scatter plot begins bivariate analysis: how two variables behave together.
Each observation contributes one point with an x-coordinate and a y-coordinate. The cloud of points can reveal positive association, negative association, nonlinear structure, clusters, outliers, heteroscedasticity, saturation, thresholds, and data errors.
Scatter plots matter because a single correlation coefficient can hide structure. Anscombe's quartet is the classic lesson: datasets can share the same means, variances, correlations, and regression line while having very different scatter plots. The visual pattern is not optional evidence; it is part of the analysis.
In AI/ML, scatter plots help with feature selection, residual analysis, calibration inspection, dimensionality-reduction displays, and sanity checks. They are often the quickest way to see whether a relationship is linear, nonlinear, absent, or driven by a few points.
# === Section 2.21: Scatter Plots ===
n = 100
fig, axes = plt.subplots(1, 3, figsize=(14, 4))
# (a) Positive correlation
x = rng.normal(50, 10, n)
y_pos = 1.5 * x + rng.normal(0, 8, n)
axes[0].scatter(x, y_pos, alpha=0.6, color="steelblue", edgecolors="white")
axes[0].set_title(f"(a) Positive Correlation\nr = {np.corrcoef(x, y_pos)[0,1]:.2f}")
# (b) No correlation
y_none = rng.normal(75, 15, n)
axes[1].scatter(x, y_none, alpha=0.6, color="#ef8a62", edgecolors="white")
axes[1].set_title(f"(b) No Correlation\nr = {np.corrcoef(x, y_none)[0,1]:.2f}")
# (c) Negative correlation
y_neg = -1.2 * x + 150 + rng.normal(0, 8, n)
axes[2].scatter(x, y_neg, alpha=0.6, color="#b2df8a", edgecolors="white")
axes[2].set_title(f"(c) Negative Correlation\nr = {np.corrcoef(x, y_neg)[0,1]:.2f}")
for ax in axes:
ax.set_xlabel("Variable X")
ax.set_ylabel("Variable Y")
plt.suptitle("Scatter Plots: Visualizing Bivariate Relationships", fontsize=13, y=1.02)
plt.tight_layout()
plt.show()
Section 2.22: Correlation¶
Correlation summarizes linear association between variables. Pearson's correlation coefficient, usually written as r, is covariance divided by the product of the two standard deviations. This normalization forces the value to lie between -1 and +1.
A positive correlation means that larger values of one variable tend to occur with larger values of the other. A negative correlation means larger values of one tend to occur with smaller values of the other. A value near zero means no linear association, not necessarily no relationship. A U-shaped or other nonlinear relationship can have correlation near zero while still being strong.
Correlation is not causation. Two variables can move together because one causes the other, because both share a hidden cause, because of selection bias, because of temporal trends, or by coincidence. Correlation is a clue, not a causal proof.
Outliers can also dominate correlation. A single high-leverage point can create or erase an apparent relationship. This is why correlation should be read with the scatter plot, not instead of it.
For categorical variables, Pearson correlation is usually not appropriate. Association among categories is commonly studied with contingency tables and chi-square statistics. The same principle holds: choose the association measure that matches the measurement scale.
# === Section 2.22: Correlation — Pearson's r ===
# Create sample data with known correlation
n = 50
x = rng.normal(100, 15, n) # e.g., hours studied
noise = rng.normal(0, 10, n)
y = 0.8 * x + 20 + noise # e.g., exam score (correlated with x)
# --- Compute Pearson's r manually ---
x_dev = x - x.mean() # deviations from mean
y_dev = y - y.mean()
numerator = np.sum(x_dev * y_dev) # sum of cross-products
denominator = (n - 1) * x.std(ddof=1) * y.std(ddof=1)
r_manual = numerator / denominator
# --- Compute with NumPy ---
r_numpy = np.corrcoef(x, y)[0, 1]
# --- Compute with SciPy (also gives p-value) ---
r_scipy, p_value = stats.pearsonr(x, y)
print("Pearson's Correlation Coefficient:")
print(f" Manual calculation: r = {r_manual:.4f}")
print(f" NumPy corrcoef: r = {r_numpy:.4f}")
print(f" SciPy pearsonr: r = {r_scipy:.4f}, p-value = {p_value:.2e}")
print(f"\n Interpretation: {'Strong' if abs(r_scipy) >= 0.7 else 'Moderate' if abs(r_scipy) >= 0.3 else 'Weak'} "
f"{'positive' if r_scipy > 0 else 'negative'} correlation")
print(f" p-value < 0.05? {p_value < 0.05} → statistically significant")
# === Correlation Heatmap — Multiple Variables at Once ===
# Seaborn makes it easy to visualize the correlation matrix
n = 200
# Simulate a small dataset with correlated features
study_hours = rng.normal(5, 2, n)
sleep_hours = rng.normal(7, 1.5, n)
exam_score = 10 * study_hours - 2 * sleep_hours + rng.normal(50, 8, n)
stress = -3 * sleep_hours + 2 * study_hours + rng.normal(20, 5, n)
df = pd.DataFrame({
"Study Hours": study_hours,
"Sleep Hours": sleep_hours,
"Exam Score": exam_score,
"Stress": stress,
})
# Correlation matrix
corr = df.corr()
print("Correlation Matrix:")
print(corr.round(3))
# Heatmap
fig, ax = plt.subplots(figsize=(7, 5))
sns.heatmap(corr, annot=True, fmt=".2f", cmap="RdBu_r", center=0,
square=True, linewidths=1, ax=ax,
vmin=-1, vmax=1)
ax.set_title("Correlation Heatmap", fontsize=13)
plt.tight_layout()
plt.show()
print("\nReading the heatmap:")
print(" Red = positive correlation, Blue = negative correlation")
print(" Darker = stronger, White = no correlation")
print(" Diagonal is always 1.0 (a variable correlates perfectly with itself)")
Section 2.23: Chapter Summary and Forward Link¶
This chapter converted raw data into interpretable information. It began with provenance and measurement scale because a statistic is meaningful only when the data source and data type support it. It then built a descriptive toolkit: center, spread, shape, quartiles, variance, standard deviation, relative variation, distribution-free bounds, box plots, histograms, quantile plots, Q-Q plots, scatter plots, and correlation.
The conceptual sequence is the important lesson. First identify what each row represents. Then identify what each column means. Then choose summaries that are valid for those measurement scales. Then visualize the distributions. Then examine outliers, skewness, variability, and relationships. Only after this work should a model enter the conversation.
Several cautions recur throughout the chapter. The mean is useful but outlier-sensitive. The median is robust but less algebraically convenient. Standard deviation is powerful but easiest to interpret under approximate normality. Chebyshev's rule is weaker but distribution-free. Correlation measures linear association, not causation. Visualization is not a luxury; it is how misleading summaries are caught.
The practical workflow for a new tabular dataset is:
- Inspect data source, row meaning, column meaning, and missingness.
- Classify each variable by measurement scale.
- Compute appropriate summaries for center, spread, shape, and frequency.
- Draw distribution plots and group comparisons.
- Investigate outliers and distributional assumptions.
- Examine bivariate relationships before modeling.
Chapter 3 extends this foundation into data visualization. The statistical summaries from this chapter tell us what to look for; visualization teaches us how to make those structures visible, honest, and persuasive.
Key Formula Reference¶
| Measure | Formula | Python |
|---|---|---|
| Mean | $\bar{x} = \dfrac{\sum x_i}{n}$ | np.mean(x) |
| Median | Middle value after sorting | np.median(x) |
| Mode | Most frequent value | stats.mode(x) |
| Range | $\max - \min$ | np.ptp(x) |
| Sample variance | $s^2 = \dfrac{\sum(x_i - \bar{x})^2}{n - 1}$ | np.var(x, ddof=1) |
| Sample standard deviation | $s = \sqrt{s^2}$ | np.std(x, ddof=1) |
| Coefficient of variation | $\dfrac{s}{\bar{x}} \times 100\%$ | stats.variation(x, ddof=1) |
| Interquartile range | $Q_3 - Q_1$ | stats.iqr(x) |
| Pearson correlation | $\dfrac{\text{Cov}(x,y)}{s_x s_y}$ | stats.pearsonr(x, y) |
| Chi-square statistic | $\sum \dfrac{(O - E)^2}{E}$ | stats.chi2_contingency(table) |
References¶
Wasserman, L. (2004). All of Statistics: A Concise Course in Statistical Inference. Springer.
Freedman, D., Pisani, R., and Purves, R. (2007). Statistics (4th ed.). W. W. Norton.
Devore, J. L. (2015). Probability and Statistics for Engineering and the Sciences (9th ed.). Cengage Learning.
NumPy Development Team. Statistics routines.
Pandas Development Team.
DataFrame.describeandDataFrame.agg.Matplotlib Developers.
matplotlib.pyplot.histand boxplot documentation.Seaborn Developers. Statistical visualization tutorial.
SciPy Development Team.
scipy.stats.