Estimators

1 min read

# topic/math/statistics

An estimator is a rule for using data to estimate an unknown population parameter.

If θ\theta is the true parameter, an estimator θ^\hat{\theta} is a statistic computed from a sample.

Examples:

  • xˉ\bar{x} estimates the population mean μ\mu
  • s2s^2 estimates the population variance σ2\sigma^2
  • p^\hat{p} estimates a population proportion pp
  • w^\hat{\mathbf{w}} in Linear Regression estimates the true regression weights

Bias:

Bias(θ^)=E[θ^]−θ\text{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta

An estimator is unbiased if its expected value equals the true parameter.

Variance: how much the estimator changes across repeated samples.

Mean squared error:

MSE(θ^)=Bias(θ^)2+Var(θ^)\text{MSE}(\hat{\theta}) = \text{Bias}(\hat{\theta})^2 + \text{Var}(\hat{\theta})

This is the statistical version of the Bias-Variance Tradeoff.

Consistency: θ^\hat{\theta} converges to θ\theta as n→∞n \to \infty.

Efficiency: among reasonable estimators, a more efficient estimator has lower variance.

Important tradeoff: an unbiased estimator is not always best. A slightly biased estimator can have lower MSE if it greatly reduces variance. This is the logic behind Regularization.

See also: Maximum Likelihood Estimation, Maximum A Posteriori Estimation, Bias-Variance Tradeoff

Linked from