Key Probability Distributions

Discrete:

Distribution	PMF	Mean	Variance	Use
Bernoulli( $p$ )	$P(X=1) = p$	$p$	$p(1-p)$	Single binary trial
Binomial( $n,p$ )	$\binom{n}{k}p^k(1-p)^{n-k}$	$np$	$np(1-p)$	Count of successes in $n$ trials
Categorical( $\mathbf{p}$ )	$P(X=k) = p_k$	—	—	One draw from $K$ classes (softmax output)

Continuous:

Distribution	PDF	Mean	Variance	Use
Gaussian( $\mu, \sigma^2$ )	$\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$	$\mu$	$\sigma^2$	Everywhere: noise, priors, weight init

Why Gaussian is everywhere:

Central limit theorem: sum of many independent RVs → Gaussian
MLE with MSE loss assumes Gaussian noise
MAP with L2 regularization = Gaussian prior on weights
Weight Initialization draws from Gaussian (or uniform) scaled by layer size

Multivariate Gaussian: $\mathcal{N}(\boldsymbol{\mu}, \Sigma)$ — fully specified by mean vector and covariance matrix. The covariance matrix $\Sigma$ must be positive semi-definite.

Key Probability Distributions

Linked from