Stochastic Gradient Descent

2 min read

# topic/math/optimization

SGD and its variants are the workhorses of neural network training.

Vanilla gradient descent: θt+1=θt−η∇L(θt)\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)

  • Uses the full dataset to compute ∇L\nabla L — expensive for large datasets

Stochastic GD: compute gradient on a single random sample. Noisy but cheap.

Mini-batch SGD (the practical default): compute gradient on a random subset (batch) of size BB:

θt+1=θt−η⋅1B∑i∈batch∇Li(θt)\theta_{t+1} = \theta_t - \eta \cdot \frac{1}{B}\sum_{i \in \text{batch}} \nabla L_i(\theta_t)

Key properties:

  • Noise from mini-batches can help escape shallow local minima and saddle points
  • Smaller batch → more noise, potential regularization effect
  • Larger batch → more stable gradients, better hardware utilization
  • The learning rate η\eta is the single most important hyperparameter → Learning Rate Schedules

Gradient clipping: before the update, compute the global norm over all gradients; if it exceeds a threshold, scale every gradient down by the same factor to bring the norm under the max. This prevents any single step from being catastrophically large (important for RNNs and transformers).

Limitations of vanilla SGD:

  • Same learning rate for all parameters
  • Struggles with ill-conditioned loss surfaces (ravines) — oscillates across the narrow direction
  • Addressed by Momentum and Adam Optimizer

See also: Gradient, Convexity, Backpropagation

Linked from