Chain Rule (Multivariable)

If $y = f(g(x))$, the chain rule gives: $\frac{dy}{dx} = \frac{df}{dg} \cdot \frac{dg}{dx}$

Multivariable generalization: if $\mathbf{y} = f(\mathbf{g}(\mathbf{x}))$ where $f: \mathbb{R}^m \to \mathbb{R}^k$ and $g: \mathbb{R}^n \to \mathbb{R}^m$:

This is a product of Jacobians: $(k \times m) \cdot (m \times n) = (k \times n)$.

Why this is everything in deep learning:

• Backpropagation is the chain rule applied to a computation graph in reverse
• For $L = \text{loss}(f_3(f_2(f_1(\mathbf{x}))))$: $\frac{\partial L}{\partial \mathbf{x}} = \frac{\partial L}{\partial f_3} \cdot \frac{\partial f_3}{\partial f_2} \cdot \frac{\partial f_2}{\partial f_1} \cdot \frac{\partial f_1}{\partial \mathbf{x}}$
• Each layer contributes one Jacobian factor; the backward pass accumulates them right-to-left

Concrete example: $z = \sigma(W\mathbf{x} + \mathbf{b})$, $L = \|z - y\|^2$

$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial (W\mathbf{x}+\mathbf{b})} \cdot \frac{\partial (W\mathbf{x}+\mathbf{b})}{\partial W}$

See also: Gradient, Backpropagation