Convolutional Neural Networks

CNNs exploit spatial structure through three key ideas: local connectivity, parameter sharing, and translation equivariance.

Convolution operation: slide a small filter (kernel) across the input, computing dot products at each position. A $3 \times 3$ filter on a $28 \times 28$ image produces a feature map highlighting where that pattern occurs.

Key components:

• Conv layer: learns $K$ filters, each producing one feature map. Parameters = $K \times C_\text{in} \times h \times w + K$ (bias)
• Pooling (MaxPool): downsamples by taking the max over local regions → spatial invariance, reduces computation
• Stride: step size of the filter; stride 2 halves spatial dimensions (alternative to pooling)
• Padding: add zeros around input to control output size

Parameter sharing: the same filter is applied at every spatial location → far fewer parameters than a fully connected layer. A $3 \times 3$ filter over 64 channels = $3 \times 3 \times 64 = 576$ params, regardless of image size.

Typical architecture: (Conv → ReLU → Pool) × N → Flatten → FC layers

Inductive Bias: CNNs assume spatial locality (nearby pixels are related) and translation invariance (a cat is a cat regardless of position). This is why they work for images but not tabular data.

See also: Inductive Bias, Activation Functions, Backpropagation