Bayesian vs Frequentist Inference

Bayesian and frequentist inference differ in how they interpret probability and unknown parameters.

Frequentist view: parameters are fixed unknown constants. Probability describes randomness in data and procedures.

Bayesian view: parameters can be treated as uncertain quantities. Probability represents degrees of belief updated by evidence.

Bayesian inference uses Bayes' theorem:

Terms:

• $P(\theta)$: prior
• $P(D \mid \theta)$: likelihood
• $P(\theta \mid D)$: posterior
• $P(D)$: evidence / normalizer

Frequentist output examples:

• Point estimate
• Confidence interval
• p-value
• Hypothesis test decision

Bayesian output examples:

• Posterior distribution
• Credible interval
• Posterior predictive distribution
• Bayes factor

Credible interval: a Bayesian interval that can be interpreted as containing the parameter with a given posterior probability, conditional on the model and prior.

Connection to ML:

• Maximum Likelihood Estimation is frequentist point estimation using likelihood
• Maximum A Posteriori Estimation is Bayesian-flavored point estimation using likelihood plus prior
• Regularization often corresponds to adding a prior over parameters

Tradeoff: Bayesian inference makes uncertainty explicit, but results can depend strongly on the prior and model assumptions.

See also: Bayes' Theorem, Maximum Likelihood Estimation, Maximum A Posteriori Estimation