Correlation and Covariance

1 min read

# topic/math/statistics

Covariance measures how two variables vary together.

Cov(X,Y)=E[(X−μX)(Y−μY)]\text{Cov}(X, Y) = \mathbb{E}[(X - \mu_X)(Y - \mu_Y)]

Sample covariance:

sxy=1n−1∑i=1n(xi−xˉ)(yi−yˉ)s_{xy} = \frac{1}{n-1}\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})

Sign:

  • Positive covariance: variables tend to move together
  • Negative covariance: one tends to rise when the other falls
  • Near zero covariance: little linear relationship

Correlation is normalized covariance:

ρXY=Cov(X,Y)σXσY\rho_{XY} = \frac{\text{Cov}(X, Y)}{\sigma_X\sigma_Y}

Correlation is unitless and lies in [−1,1][-1, 1].

Covariance matrix: for a vector-valued random variable, the covariance matrix stores variances on the diagonal and covariances off the diagonal.

Σij=Cov(Xi,Xj)\Sigma_{ij} = \text{Cov}(X_i, X_j)

Covariance matrices are central to Principal Component Analysis (PCA) and Gaussian models.

Limitations:

  • Correlation measures linear association, not all dependence
  • Correlation does not imply causation
  • Outliers can dominate correlation
  • Restricted ranges can hide relationships

See also: Expectation and Variance, Principal Component Analysis (PCA), Experimental Design

Linked from