Sampling Distributions

A sampling distribution is the distribution of a statistic across repeated samples from the same population.

Example: take many samples of size $n$, compute the sample mean $\bar{x}$ for each one, then plot those means. That distribution is the sampling distribution of $\bar{x}$.

Why it matters: statistical inference needs to know how much an estimate would vary if the data collection were repeated.

For iid samples with population mean $\mu$ and variance $\sigma^2$:

The standard deviation of an estimator's sampling distribution is its standard error.

For the sample mean:

If $\sigma$ is unknown, estimate it with:

Key implication: uncertainty shrinks like $1/\sqrt{n}$, not $1/n$. Getting 10x lower standard error requires about 100x more data.

In ML: validation metrics also have sampling distributions. A model with 91.0% accuracy is not automatically better than one with 90.8% accuracy unless the difference is large relative to evaluation uncertainty.

See also: Population vs Sample, Central Limit Theorem, Confidence Intervals