Neural Networks

Neural networks learn composed functions from data. The useful mental model is: layers apply differentiable transformations, nonlinearities make the composition expressive, and backpropagation assigns credit to parameters through the computation graph.

Core sequence:

1. The Artificial Neuron - the basic affine-plus-nonlinearity unit.
2. Activation Functions - nonlinearities that make stacked layers more expressive than one linear map.
3. Softmax - turns logits into a categorical distribution.
4. Backpropagation - computes gradients through the network using the chain rule.
5. Weight Initialization - sets initial parameter scales so signals and gradients do not explode or vanish immediately.
6. Batch Normalization - normalizes activations using batch statistics.
7. Layer Normalization - normalizes features within each example, common in transformers.
8. Dropout - randomly masks activations during training as regularization.
9. Residual Connections - add skip paths that make deep networks easier to optimize.
10. Convolutional Neural Networks - use local filters and weight sharing for spatial data.
11. Recurrent Neural Networks - process sequences through recurrent state.
12. LSTM and GRU - gated recurrent architectures that reduce vanishing-gradient problems.
13. Embeddings - learned vector representations for discrete objects.

How the pieces fit:

• Neurons, activations, and softmax define the forward computation.
• Backpropagation and optimization define how parameters change.
• Initialization, normalization, dropout, and residual connections stabilize or regularize training.
• CNNs and RNNs add architectural bias for images and sequences.
• Embeddings let discrete tokens, categories, or IDs participate in continuous optimization.

Core equations to keep active:

• Neuron: $h = \phi(\mathbf{w}^\top x + b)$
• Softmax: $p_i = \frac{\exp z_i}{\sum_j \exp z_j}$
• Cross-entropy loss: $L = -\sum_i y_i \log p_i$
• Backpropagation pattern: $\frac{\partial L}{\partial w} = \frac{\partial L}{\partial h}\frac{\partial h}{\partial w}$
• Batch normalization: $\hat{x} = \frac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}$
• Residual block: $y = F(x) + x$
• Dropout mask: $\tilde{h} = m \odot h$ with $m_i \sim \mathrm{Bernoulli}(p)$
• Embedding lookup: $e_i = E[i]$

See also: Calculus and Optimization, Classical Machine Learning, Transformers and LLMs