# topic/math/optimization

Adam Optimizer

Adam (Adaptive Moment Estimation) combines Momentum with per-parameter adaptive learning rates:

Calculus and Optimization

Calculus studies local change; optimization uses local change to choose better parameters. The useful mental model is: derivatives describe sensitivity, Taylor expansions…

13 links · 2 backlinks

# topic/math/calculus# topic/math/optimization

Computation Graphs

A computation graph is a directed acyclic graph (DAG) where nodes are operations and edges carry values. It makes the Chain Rule (Multivariable) systematic.

3 links · 4 backlinks

# topic/math/optimization# topic/cs

Convexity

A function is convex if a line segment between any two points on its graph lies above the graph:

4 links · 7 backlinks

# topic/math/optimization

Learning Rate Schedules

The learning rate controls step size in gradient descent. Too large → divergence; too small → slow convergence. Schedules vary during training.

2 links · 4 backlinks

# topic/math/optimization

Momentum

Momentum accelerates Stochastic Gradient Descent by accumulating a velocity vector in directions of persistent gradient:

2 links · 3 backlinks

# topic/math/optimization

Stochastic Gradient Descent

SGD and its variants are the workhorses of neural network training.

6 links · 12 backlinks

# topic/math/optimization