Taylor Expansion

1 min read

# topic/math/calculus

Taylor expansion approximates a function near a point using its derivatives:

1D: f(x+δ)≈f(x)+f′(x)δ+12f′′(x)δ2+…f(x + \delta) \approx f(x) + f'(x)\delta + \frac{1}{2}f''(x)\delta^2 + \dots

Multivariable (2nd order):

f(x+δ)≈f(x)+∇f(x)⊤δ+12δ⊤H(x)δf(\mathbf{x} + \delta) \approx f(\mathbf{x}) + \nabla f(\mathbf{x})^\top\delta + \frac{1}{2}\delta^\top H(\mathbf{x})\delta

where ∇f\nabla f is the Gradient and HH is the Hessian Matrix.

Why it matters in ML:

  • Gradient descent uses the 1st-order approximation: step in direction −∇f-\nabla f
  • Newton's method uses the 2nd-order approximation: set the gradient of the quadratic to zero → δ∗=−H−1∇f\delta^* = -H^{-1}\nabla f
  • Loss surface analysis — the 2nd-order term reveals whether a critical point is a minimum, maximum, or saddle (positive definiteness of H)
  • Natural gradient and Fisher information also derive from 2nd-order Taylor of KL divergence

The quality of the approximation depends on how smooth ff is and how large δ\delta is. Neural network loss surfaces are highly non-quadratic globally, but locally the quadratic approximation motivates most optimization theory.

See also: Convexity, Learning Rate Schedules

Linked from