Momentum

1 min read

# topic/math/optimization

Momentum accelerates Stochastic Gradient Descent by accumulating a velocity vector in directions of persistent gradient:

vt=βvt−1+∇L(θt)v_t = \beta v_{t-1} + \nabla L(\theta_t) θt+1=θt−ηvt\theta_{t+1} = \theta_t - \eta v_t

Typical β=0.9\beta = 0.9.

Intuition: a ball rolling downhill accumulates speed. Gradients that consistently point the same direction build up velocity; oscillating gradients cancel out.

Benefits:

  • Dampens oscillations across narrow ravines
  • Accelerates movement along consistent gradient directions
  • Helps escape shallow local minima and saddle points

Nesterov momentum (lookahead variant):

vt=βvt−1+∇L(θt−ηβvt−1)v_t = \beta v_{t-1} + \nabla L(\theta_t - \eta\beta v_{t-1})

Evaluates the gradient at the "future" position θ−ηβv\theta - \eta\beta v rather than the current position. This corrective step often converges faster.

Relation to Adam: Adam Optimizer combines momentum (first moment) with per-parameter adaptive learning rates (second moment). Momentum is the "m" in Adam.

See also: Stochastic Gradient Descent, Adam Optimizer

Linked from