Law of Large Numbers

The Law of Large Numbers says that the sample average converges to the expected value as sample size grows.

For iid random variables $X_1, \dots, X_n$ with finite mean $\mu$:

as $n \to \infty$.

Intuition: random fluctuations cancel out when you average enough independent observations.

What it does say:

• Larger samples make averages more stable
• Empirical frequencies converge to probabilities
• Monte Carlo estimates improve with more samples

What it does not say:

• Small samples are reliable
• Any single observation gets closer to the mean
• Biased samples become unbiased as they grow

The independence and identical-distribution assumptions matter. More data from the wrong distribution can make you more confident in the wrong answer.

In ML: empirical risk minimization depends on this idea. Training loss estimates expected loss only when examples are representative of the target distribution.

See also: Random Variables, Expectation and Variance, Central Limit Theorem