Gradient

The gradient $\nabla f$ of a scalar function $f : \mathbb{R}^n \to \mathbb{R}$ is the vector of all partial derivatives:

Key properties:

• Points in the direction of steepest ascent of $f$
• Its magnitude $\|\nabla f\|$ is the rate of steepest ascent
• It is perpendicular to level sets (contour surfaces) of $f$
• $\nabla f = 0$ at critical points (local min, max, or saddle)

Gradient descent walks downhill: $\mathbf{x}_{t+1} = \mathbf{x}_t - \eta \nabla f(\mathbf{x}_t)$, where $\eta$ is the learning rate. This is the foundation of all neural network training via Stochastic Gradient Descent.

Numerical gradient (finite differences): $\frac{\partial f}{\partial x_i} \approx \frac{f(x + \epsilon e_i) - f(x - \epsilon e_i)}{2\epsilon}$. Useful for gradient checking but $O(n)$ cost per evaluation.

See also: Chain Rule (Multivariable), Jacobian Matrix, Hessian Matrix