Computation Graphs

A computation graph is a directed acyclic graph (DAG) where nodes are operations and edges carry values. It makes the Chain Rule (Multivariable) systematic.

Forward pass: evaluate the graph from inputs to output, storing intermediate values at each node.

Backward pass (backprop): propagate gradients from output to inputs:

• Each node computes its local gradient (derivative of output w.r.t. its inputs)
• Multiply by the upstream gradient (from the loss) via the chain rule
• This is Backpropagation

Example: $L = (y - \sigma(w_1 x + w_2))^2$

x → [×w1] → [+w2] → [σ] → [-y] → [²] → L

Forward: compute left-to-right, store each intermediate. Backward: compute $\partial L/\partial$ each node right-to-left.

Key properties:

• Automatic differentiation frameworks (PyTorch, JAX) build computation graphs dynamically or statically
• Memory cost: must store all intermediate values for backward pass (activation checkpointing trades compute for memory)
• Multiple paths from input to output → gradients sum (multivariate chain rule)

This is the foundation of all modern deep learning: loss.backward() in PyTorch traverses the computation graph backward, computing gradients for every parameter.

See also: Backpropagation, Chain Rule (Multivariable), Gradient