Probability

Probability studies uncertainty before observing data. The useful mental model is: random variables turn outcomes into quantities, distributions assign mass or density, expectations summarize long-run behavior, and conditioning updates beliefs when information arrives.

Core sequence:

1. Random Variables - the basic objects. A random variable maps outcomes to numbers and carries a probability distribution.
2. Key Probability Distributions - reusable models for uncertainty: Bernoulli, binomial, categorical, Gaussian, exponential, and related families.
3. Expectation and Variance - summarize a distribution by its center and spread. These are the quantities most learning algorithms optimize or estimate.
4. Bayes' Theorem - updates prior beliefs with evidence through likelihoods. This is the algebra of conditioning.
5. Maximum Likelihood Estimation - chooses parameters that make observed data most probable under a model.
6. Maximum A Posteriori Estimation - combines likelihood with a prior. It is regularized estimation under a probabilistic interpretation.
7. Entropy and Cross-Entropy - measures uncertainty and prediction mismatch in distributions.
8. KL Divergence - measures directed mismatch between two distributions. It is central to variational inference, RLHF, and policy regularization.

How the pieces fit:

• Random variables define what can vary.
• Distributions define how likely each value or region is.
• Expectation and variance summarize distributions into usable quantities.
• Conditioning and Bayes' theorem describe how evidence changes probabilities.
• MLE and MAP turn probabilistic models into parameter estimates.
• Entropy, cross-entropy, and KL divergence compare distributions and become losses.

Core equations to keep active:

• Probability mass normalization: $\sum_x p(x) = 1$
• Probability density normalization: $\int p(x)dx = 1$
• Expectation: $\mathbb{E}[X] = \sum_x x p(x)$ or $\mathbb{E}[X] = \int x p(x)dx$
• Variance: $\mathrm{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2]$
• Conditional probability: $p(x|y) = \frac{p(x,y)}{p(y)}$
• Bayes' theorem: $p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)}$
• MLE: $\hat{\theta}_{\mathrm{MLE}} = \arg\max_\theta \sum_i \log p(x_i|\theta)$
• MAP: $\hat{\theta}_{\mathrm{MAP}} = \arg\max_\theta \left[\sum_i \log p(x_i|\theta) + \log p(\theta)\right]$
• Entropy: $H(p) = -\sum_x p(x)\log p(x)$
• KL divergence: $D_{\mathrm{KL}}(p \| q) = \sum_x p(x)\log\frac{p(x)}{q(x)}$

See also: Statistics Fundamentals