# topic/math/probability

Bayes' Theorem

Components: - — posterior: updated belief about after observing - — likelihood: probability of evidence given - — prior: belief about before seeing evidence - — marginal…

Entropy and Cross-Entropy

Entropy measures the average surprise (information content) of a distribution:

Expectation and Variance

Expectation (mean): the average value of a random variable. - Discrete: - Continuous: - Linearity: (always, even if dependent)

Key Probability Distributions

Discrete:

KL Divergence

KL divergence measures how one probability distribution diverges from a reference distribution :

Maximum A Posteriori Estimation

MAP estimation adds a prior to Maximum Likelihood Estimation :

Maximum Likelihood Estimation

MLE finds the parameters that maximize the probability of the observed data:

Probability

Probability studies uncertainty before observing data. The useful mental model is: random variables turn outcomes into quantities, distributions assign mass or density…

Random Variables

A random variable is a function from outcomes to numbers, equipped with a probability distribution describing the likelihood of each value.