Statistics Fundamentals

Statistics studies inference from finite data. The useful mental model is: probability goes from model to possible data; statistics goes from observed data back to uncertain claims about the model, population, or intervention.

Core sequence:

1. Population vs Sample - separates the true target from the finite data observed.
2. Estimators - functions of data used to approximate unknown quantities.
3. Sampling Distributions - distributions of estimators across repeated samples.
4. Law of Large Numbers - explains why averages stabilize as sample size grows.
5. Central Limit Theorem - explains why many estimator distributions become approximately Gaussian.
6. Confidence Intervals - quantify estimator uncertainty under repeated sampling.
7. Hypothesis Testing - evaluates how surprising observed data is under a null model.
8. Statistical Power - measures the chance a test detects a real effect.
9. Statistical Significance vs Practical Significance - separates detectable effects from meaningful effects.
10. Correlation and Covariance - measures linear association and joint variation.
11. Experimental Design - controls data collection so causal or comparative claims are defensible.
12. Bootstrap and Resampling - estimates uncertainty by resampling observed data.
13. Bayesian vs Frequentist Inference - compares two interpretations of probability and uncertainty.

How the pieces fit:

• Populations define the target; samples provide finite evidence.
• Estimators compress samples into guesses about unknown quantities.
• Sampling distributions, LLN, and CLT describe estimator behavior.
• Confidence intervals, hypothesis tests, and power turn estimator behavior into uncertainty statements.
• Correlation, covariance, and experimental design determine which claims are descriptive and which can support causal interpretation.
• Bootstrap methods approximate uncertainty when analytic sampling distributions are hard.
• Bayesian and frequentist views differ in how they treat unknown parameters and probability.

Core equations to keep active:

• Sample mean: $\bar{x} = \frac{1}{n}\sum_i x_i$
• Sample variance: $s^2 = \frac{1}{n-1}\sum_i (x_i - \bar{x})^2$
• Standard error of the mean: $\mathrm{SE}(\bar{x}) = \frac{s}{\sqrt{n}}$
• Approximate confidence interval: $\hat{\theta} \pm z_{\alpha/2}\mathrm{SE}(\hat{\theta})$
• Test statistic pattern: $z = \frac{\hat{\theta} - \theta_0}{\mathrm{SE}(\hat{\theta})}$
• Power: $\mathrm{power} = 1 - \beta$
• Correlation: $r = \frac{\mathrm{cov}(X,Y)}{s_X s_Y}$
• Bootstrap estimator: compute $\hat{\theta}^{*b}$ over resampled datasets $b = 1,\dots,B$

See also: Probability, Linear Regression, Evaluation Metrics