Positive Definite Matrices

A symmetric matrix $A$ is positive definite (PD) if $\mathbf{x}^\top A\mathbf{x} > 0$ for all nonzero $\mathbf{x}$. Positive semi-definite (PSD) allows $\geq 0$.

Equivalent characterizations (any one implies the others):

• All eigenvalues are positive ($> 0$ for PD, $\geq 0$ for PSD)
• All leading principal minors are positive
• $A = B^\top B$ for some full-rank matrix $B$
• The quadratic form $f(\mathbf{x}) = \mathbf{x}^\top A\mathbf{x}$ has a unique global minimum at $\mathbf{x} = 0$

Quadratic forms: $f(\mathbf{x}) = \mathbf{x}^\top A\mathbf{x}$ defines a bowl shape when $A$ is PD (like a paraboloid). The eigenvectors of $A$ are the principal axes and eigenvalues are the curvatures along those axes.

Where it appears:

• Covariance matrices are always PSD — $\Sigma = \mathbb{E}[(\mathbf{x}-\mu)(\mathbf{x}-\mu)^\top]$
• Hessian Matrix — if the Hessian is PD at a point, that point is a local minimum
• Optimization — PD Hessian means the loss surface is locally convex → Convexity
• Kernel matrices (Gram matrices) must be PSD

See also: Eigendecomposition , Taylor Expansion