Hessian Matrix

The Hessian $H$ of a scalar function $f: \mathbb{R}^n \to \mathbb{R}$ is the $n \times n$ matrix of second partial derivatives:

The Hessian is symmetric (by Schwarz's theorem: mixed partials are equal).

What it tells you: curvature of the function surface at a point.

Critical point classification (where $\nabla f = 0$):

• $H$ is positive definite → local minimum
• $H$ is negative definite → local maximum
• $H$ has both positive and negative eigenvalues → saddle point

Connection to Taylor Expansion:

The $\frac{1}{2}\delta^\top H\delta$ term captures the curvature. Large eigenvalues of $H$ mean sharp curvature (the loss changes rapidly); small eigenvalues mean flat directions.

In optimization:

• Newton's method uses $H^{-1}$: $\mathbf{x}_{t+1} = \mathbf{x}_t - H^{-1}\nabla f$ — converges faster than gradient descent but $O(n^2)$ storage, $O(n^3)$ inversion
• The condition number $\kappa(H) = \lambda_{\max}/\lambda_{\min}$ governs how poorly gradient descent performs (elongated loss valleys)
• Adam Optimizer partially addresses ill-conditioning by adapting per-parameter learning rates

See also: Gradient, Convexity