Calculus and Optimization

Calculus studies local change; optimization uses local change to choose better parameters. The useful mental model is: derivatives describe sensitivity, Taylor expansions approximate functions locally, and optimizers use those local signals to move through a loss surface.

Core sequence:

1. Taylor Expansion - approximates a function near a point using derivatives.
2. Gradient - gives the direction of steepest local increase for scalar-valued functions.
3. Chain Rule (Multivariable) - composes derivatives through nested functions.
4. Jacobian Matrix - organizes first derivatives for vector-valued functions.
5. Hessian Matrix - organizes second derivatives and captures local curvature.
6. Convexity - identifies problems where local minima are global minima.
7. Computation Graphs - represents composed functions so derivatives can be propagated mechanically.
8. Stochastic Gradient Descent - updates parameters using noisy gradient estimates from minibatches.
9. Momentum - smooths update directions by accumulating velocity.
10. Adam Optimizer - adapts per-parameter learning rates using first and second moment estimates.
11. Learning Rate Schedules - controls step size over training.

How the pieces fit:

• Taylor expansion is the local approximation that makes gradient methods sensible.
• Gradients, Jacobians, and Hessians expose first-order and second-order structure.
• The chain rule and computation graphs are the basis of backpropagation.
• Convexity gives guarantees; nonconvex neural networks rely on heuristics and scale instead.
• SGD, momentum, Adam, and schedules are practical rules for turning derivatives into parameter updates.

Core equations to keep active:

• First-order Taylor approximation: $f(x+\Delta x) \approx f(x) + \nabla f(x)^\top \Delta x$
• Second-order Taylor approximation: $f(x+\Delta x) \approx f(x) + \nabla f(x)^\top \Delta x + \frac{1}{2}\Delta x^\top H \Delta x$
• Gradient descent: $\theta_{t+1} = \theta_t - \eta \nabla_\theta L(\theta_t)$
• Multivariable chain rule: $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y}\frac{\partial y}{\partial x}$
• Jacobian: $J_{ij} = \frac{\partial f_i}{\partial x_j}$
• Hessian: $H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}$
• Convexity: $f(\lambda x + (1-\lambda)y) \leq \lambda f(x) + (1-\lambda)f(y)$
• Momentum: $v_t = \beta v_{t-1} + \nabla_\theta L(\theta_t)$, $\theta_{t+1} = \theta_t - \eta v_t$
• Adam update direction: $\theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$

See also: Backpropagation, Linear Algebra