Maximum Likelihood Estimation

1 min read

# topic/math/probability

MLE finds the parameters θ\theta that maximize the probability of the observed data:

θ^MLE=arg⁡max⁡θP(data∣θ)=arg⁡max⁡θ∏i=1np(xi∣θ)\hat{\theta}_{\text{MLE}} = \arg\max_\theta P(\text{data} | \theta) = \arg\max_\theta \prod_{i=1}^n p(x_i | \theta)

In practice, maximize the log-likelihood (log turns product into sum):

θ^MLE=arg⁡max⁡θ∑i=1nlog⁡p(xi∣θ)\hat{\theta}_{\text{MLE}} = \arg\max_\theta \sum_{i=1}^n \log p(x_i | \theta)

Key insight: minimizing cross-entropy loss is maximizing log-likelihood.

  • For classification: CE=−∑yilog⁡y^i\text{CE} = -\sum y_i \log \hat{y}_i — this is the negative log-likelihood of the categorical distribution
  • For regression with MSE loss: equivalent to MLE assuming Gaussian noise

MLE for a Gaussian:

  • μ^=1n∑xi\hat{\mu} = \frac{1}{n}\sum x_i (sample mean)
  • σ^2=1n∑(xi−μ^)2\hat{\sigma}^2 = \frac{1}{n}\sum(x_i - \hat{\mu})^2 (sample variance, biased)

Properties:

  • Consistent — converges to true θ\theta as n→∞n \to \infty
  • Asymptotically efficient — achieves lowest variance among consistent estimators
  • Can overfit — maximizing likelihood on training data doesn't penalize complexity → motivates Regularization and Maximum A Posteriori Estimation

See also: Bayes' Theorem, Loss Functions

Linked from