Learning Rate Schedules

The learning rate $\eta$ controls step size in gradient descent. Too large → divergence; too small → slow convergence. Schedules vary $\eta$ during training.

Common schedules:

Step decay: reduce $\eta$ by a factor every $N$ epochs. Simple but requires manual tuning.

Exponential decay: $\eta_t = \eta_0 \cdot \gamma^t$. Smooth version of step decay.

Cosine annealing: $\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})(1 + \cos(\pi t/T))$

• Smooth decay from $\eta_{\max}$ to $\eta_{\min}$ following a cosine curve
• Standard for transformer training

Warmup: start with very small $\eta$ and linearly increase for the first $N$ steps.

• Prevents early instability when model weights are random and gradients are large
• Critical for transformers — common recipe: linear warmup → cosine decay

1cycle policy: warmup to peak, then anneal below initial LR. Good for fast convergence.

Why schedules help:

• Large $\eta$ early → explore broadly, escape bad local minima
• Small $\eta$ later → fine-tune, settle into a good minimum
• Warmup stabilizes early training when loss landscape is poorly conditioned

See also: Stochastic Gradient Descent, Adam Optimizer