Maximum A Posteriori Estimation

1 min read

# topic/math/probability

MAP estimation adds a prior to Maximum Likelihood Estimation :

θ^MAP=arg⁡max⁡θP(θ∣data)=arg⁡max⁡θP(data∣θ)⋅P(θ)\hat{\theta}_{\text{MAP}} = \arg\max_\theta P(\theta | \text{data}) = \arg\max_\theta P(\text{data}|\theta) \cdot P(\theta)

Taking logs: θ^∗MAP=arg⁡max⁡∗θ[∑ilog⁡p(xi∣θ)+log⁡p(θ)]\hat{\theta}*{\text{MAP}} = \arg\max*\theta \left[\sum_i \log p(x_i|\theta) + \log p(\theta)\right]

The prior acts as a regularizer:

PriorPenaltyEffect
Gaussian N(0,σ2)\mathcal{N}(0, \sigma^2)−λ2∥θ∥2-\frac{\lambda}{2}\lVert\theta\rVert^2 (L2)Weight decay — shrinks all weights toward zero
Laplace−λ∥θ∥1-\lambda\lVert\theta\rVert_1 (L1)Sparsity — drives some weights exactly to zero

Key insight: L2 Regularization = MAP with Gaussian prior. L1 regularization = MAP with Laplace prior. The regularization strength λ\lambda is inversely proportional to the prior variance.

MAP vs MLE vs Full Bayesian:

  • MLE — no prior, just likelihood → prone to overfitting
  • MAP — point estimate with prior → regularized, but still a single point
  • Full Bayesian — compute entire posterior P(θ∣data)P(\theta|\text{data}) → most principled but often intractable

See also: Bayes' Theorem , Regularization , Bias-Variance Tradeoff

Linked from