Jacobian Matrix

2 min read

# topic/math/calculus

The Jacobian of a vector-valued function f:Rn→Rm\mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m is the m×nm \times n matrix of all partial derivatives:

J=[∂f1∂x1⋯∂f1∂xn⋮⋱⋮∂fm∂x1⋯∂fm∂xn]J = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{bmatrix}

Intuition: the Jacobian is the best linear approximation of f\mathbf{f} near a point: f(x+δ)≈f(x)+Jδ\mathbf{f}(\mathbf{x} + \delta) \approx \mathbf{f}(\mathbf{x}) + J\delta.

Special cases:

  • When m=1m = 1 (scalar output): Jacobian reduces to the Gradient (as a row vector)
  • When n=1n = 1 (scalar input): Jacobian is the derivative vector

In deep learning: each layer hl+1=fl(hl)\mathbf{h}_{l+1} = f_l(\mathbf{h}_l) has a Jacobian Jl=∂hl+1∂hlJ_l = \frac{\partial \mathbf{h}_{l+1}}{\partial \mathbf{h}_l}. Backpropagation multiplies these Jacobians via the Chain Rule (Multivariable):

∂L∂hl=∂L∂hl+1⋅Jl\frac{\partial L}{\partial \mathbf{h}_l} = \frac{\partial L}{\partial \mathbf{h}_{l+1}} \cdot J_l

If Jacobian singular values are consistently <1< 1 → vanishing gradients. If >1> 1 → exploding gradients.

Useful Jacobians (the building blocks of backprop):

∂∂x(Wx+b)=W,∂∂b(Wx+b)=I,∂∂zf(z)=diag⁡(f′(z)),∂∂u(u⊤h)=h⊤\frac{\partial}{\partial \mathbf{x}}(W\mathbf{x} + \mathbf{b}) = W, \quad \frac{\partial}{\partial \mathbf{b}}(W\mathbf{x} + \mathbf{b}) = I, \quad \frac{\partial}{\partial \mathbf{z}} f(\mathbf{z}) = \operatorname{diag}(f'(\mathbf{z})), \quad \frac{\partial}{\partial \mathbf{u}}(\mathbf{u}^\top \mathbf{h}) = \mathbf{h}^\top

An element-wise activation has a diagonal Jacobian because hi=f(zi)h_i = f(z_i) depends only on ziz_i.

Jacobian determinant: ∣det⁡(J)∣|\det(J)| measures how the transformation locally scales volumes (used in normalizing flows and change of variables for probability distributions).

See also: Hessian Matrix, Gradient

Linked from