The Artificial Neuron

The fundamental unit of a neural network:

1. Linear transformation: $z = \mathbf{w}^\top\mathbf{x} + b$ — weighted sum of inputs plus bias
2. Nonlinear activation: $a = \sigma(z)$ — applies an activation function

Without the nonlinearity, stacking layers would collapse to a single linear transformation (composition of linear maps is linear). The activation function is what makes depth useful.

A single neuron with sigmoid activation = Logistic Regression.

Multi-layer perceptron (MLP): stack layers of neurons:

Universal approximation theorem: a single hidden layer with enough neurons can approximate any continuous function. But depth is more parameter-efficient than width in practice.

Batched form: in practice a whole batch of $m$ inputs is processed at once by stacking them as rows of $X \in \mathbb{R}^{m \times d_\text{in}}$, so a layer is $H = f(XW + \mathbf{b})$ with $W \in \mathbb{R}^{d_\text{in} \times d_\text{out}}$ and $\mathbf{b}$ broadcast across rows.

PyTorch detail: nn.Linear stores its weight as $d_\text{out} \times d_\text{in}$ and computes X @ W.T. The transpose is free (it only changes the stride), and storing it this way makes the gradient $\partial L/\partial W$ come out already shaped like $W$.

Parameters per layer: for input dim $d_\text{in}$, output dim $d_\text{out}$: $d_\text{in} \times d_\text{out}$ weights + $d_\text{out}$ biases.

See also: Activation Functions, Backpropagation, Loss Functions