Activation Functions

Activation functions introduce nonlinearity after the linear transformation in each neuron.

Function	Formula	Range	Notes
Sigmoid	$\frac{1}{1+e^{-x}}$	$(0, 1)$	Historical; saturates → vanishing gradients
Tanh	$\frac{e^x - e^{-x}}{e^x + e^{-x}}$	$(-1, 1)$	Zero-centered but still saturates
ReLU	$\max(0, x)$	$[0, \infty)$	Default. Fast, no saturation for $x > 0$ . Dead neurons if $x < 0$ always
Leaky ReLU	$\max(\alpha x, x)$	$(-\infty, \infty)$	Fixes dead neuron problem, $\alpha \sim 0.01$
GELU	$x \cdot \Phi(x)$	$\approx(-0.17, \infty)$	Smooth ReLU. Default in transformers (BERT, GPT)
SwiGLU	$\text{Swish}(xW) \cdot (xV)$	—	Gated variant. Used in modern LLMs (LLaMA, PaLM)

Why ReLU dominates:

Gradient is 1 for positive inputs → no vanishing gradient
Computationally trivial (just a threshold)
Sparse activation (many zeros) → efficient representation
Dying ReLU: if a pre-activation is negative for every input, it receives zero gradient forever and the unit goes dead. Leaky ReLU ( $\alpha x$ for $x \leq 0$ ) fixes this.

Gated activations (modern LLMs):

Swish (smooth, non-monotonic): $\text{Swish}(x) = x \cdot \sigma(x)$
GLU — one projection produces content, another produces a gate: $\text{GLU}(x) = xW_1 \odot \sigma(xW_2)$
SwiGLU plugs Swish into the GLU gate: $\text{SwiGLU}(x) = (xW_1) \odot \text{Swish}(xW_2)$

Swish derivative reuses the sigmoid derivative: $\frac{\partial}{\partial x}\text{Swish}(x) = \sigma(x) + \text{Swish}(x)(1 - \sigma(x))$ .

Why non-linearities are essential: without them, stacked layers collapse to a single linear map ( $W_1 W_2 \mathbf{x} = W\mathbf{x}$ ) — extra depth adds no representational power. With non-linearities, deep networks become universal function approximators.

Choosing:

CNNs → ReLU
Transformers → GELU or SwiGLU
Output layer: sigmoid (binary), softmax (multiclass), none (regression)

Activation Functions

Linked from