# topic/math

Adam Optimizer

Adam (Adaptive Moment Estimation) combines Momentum with per-parameter adaptive learning rates:

Bayes' Theorem

Components: - — posterior: updated belief about after observing - — likelihood: probability of evidence given - — prior: belief about before seeing evidence - — marginal…

Bayesian vs Frequentist Inference

Bayesian and frequentist inference differ in how they interpret probability and unknown parameters.

Bootstrap and Resampling

The bootstrap estimates uncertainty by resampling from the observed data.

Calculus and Optimization

Calculus studies local change; optimization uses local change to choose better parameters. The useful mental model is: derivatives describe sensitivity, Taylor expansions…

Central Limit Theorem

The Central Limit Theorem says that sums and averages of many independent random variables become approximately normal under broad conditions.

Chain Rule (Multivariable)

If , the chain rule gives:

Closure

A set is closed under an operation if applying the operation to elements of always produces an element of :

Computation Graphs

A computation graph is a directed acyclic graph (DAG) where nodes are operations and edges carry values. It makes the Chain Rule (Multivariable) systematic.

Confidence Intervals

A confidence interval is a procedure that produces a range of plausible parameter values from data.

Convexity

A function is convex if a line segment between any two points on its graph lies above the graph:

Correlation and Covariance

Covariance measures how two variables vary together.

Cross Product

The cross product is defined only in and and produces a vector (unlike the Dot Product which produces a scalar):

Determinant and Inverse

The determinant of a square matrix is a scalar that captures how the transformation scales volume:

Dot Product

The dot product (inner product) of two vectors can be understood through two equivalent views:

Eigendecomposition

An eigenvector of matrix is a nonzero vector whose direction is unchanged by the transformation: , where is the eigenvalue.

Entropy and Cross-Entropy

Entropy measures the average surprise (information content) of a distribution:

Estimators

An estimator is a rule for using data to estimate an unknown population parameter.

Expectation and Variance

Expectation (mean): the average value of a random variable. - Discrete: - Continuous: - Linearity: (always, even if dependent)

Experimental Design

Experimental design is about collecting data so comparisons support valid causal or statistical conclusions.

Gaussian Elimination

Gaussian elimination transforms a matrix into row echelon form (REF) using elementary row operations to solve linear systems .

Gradient

The gradient of a scalar function is the vector of all partial derivatives:

Hessian Matrix

The Hessian of a scalar function is the matrix of second partial derivatives:

Hypothesis Testing

A hypothesis test asks whether observed data is surprising under a null hypothesis.

Jacobian Matrix

The Jacobian of a vector-valued function is the matrix of all partial derivatives:

Key Probability Distributions

Discrete:

KL Divergence

KL divergence measures how one probability distribution diverges from a reference distribution :

Law of Large Numbers

The Law of Large Numbers says that the sample average converges to the expected value as sample size grows.

Learning Rate Schedules

The learning rate controls step size in gradient descent. Too large → divergence; too small → slow convergence. Schedules vary during training.

Linear Algebra

Linear algebra studies vector spaces and linear maps between them. The useful mental model is: vectors live in spaces, bases give coordinates, matrices represent transformations…

Linear Transformations

A linear transformation satisfies . Every linear transformation can be represented as multiplication by a matrix , and every matrix defines one.

LoRA

LoRA (Low-Rank Adaptation) makes fine-tuning large models practical by training only small low-rank matrices.

Matrix Multiplication

Matrix multiplication can be understood through three equivalent views:

Maximum A Posteriori Estimation

MAP estimation adds a prior to Maximum Likelihood Estimation :

Maximum Likelihood Estimation

MLE finds the parameters that maximize the probability of the observed data:

Momentum

Momentum accelerates Stochastic Gradient Descent by accumulating a velocity vector in directions of persistent gradient:

Norms and Distance Metrics

A norm measures the "size" of a vector. Norms underlie nearly every loss function, regularizer, and similarity measure in ML.

Orthogonality and Projections

Two vectors are orthogonal if . An orthonormal set has all vectors mutually orthogonal with unit length.

Population vs Sample

A population is the full data-generating group you care about. A sample is the observed subset used to infer properties of that population.

Positive Definite Matrices

A symmetric matrix is positive definite (PD) if for all nonzero . Positive semi-definite (PSD) allows .

Power Iteration

Power iteration is a simple algorithm to find the dominant eigenvalue (largest in absolute value) and its eigenvector.

Principal Component Analysis (PCA)

PCA finds the directions of maximum variance in data and projects onto them for dimensionality reduction.

Probability

Probability studies uncertainty before observing data. The useful mental model is: random variables turn outcomes into quantities, distributions assign mass or density…

Random Variables

A random variable is a function from outcomes to numbers, equipped with a probability distribution describing the likelihood of each value.

Rank and Null Space

For a matrix ( ):

Sampling Distributions

A sampling distribution is the distribution of a statistic across repeated samples from the same population.

Singular Value Decomposition (SVD)

Every matrix (any shape) can be decomposed as :

Statistical Power

Statistical power is the probability that a test correctly detects a real effect.

Statistical Significance vs Practical Significance

Statistical significance asks whether an observed effect is unlikely under a null hypothesis.

Statistics Fundamentals

Statistics studies inference from finite data. The useful mental model is: probability goes from model to possible data; statistics goes from observed data back to uncertain…

Stochastic Gradient Descent

SGD and its variants are the workhorses of neural network training.

Taylor Expansion

Taylor expansion approximates a function near a point using its derivatives:

Vector Spaces and Basis

A vector space over is a set of vectors closed under addition and scalar multiplication. is the canonical example.