# topic/ai-ml

Activation Functions

Activation functions introduce nonlinearity after the linear transformation in each neuron.

Actor-Critic Methods

Actor-critic combines a policy (actor) with a value function (critic) for stable, low-variance policy gradient learning.

Advantage Function

The advantage function measures how much better an action is compared to the average action in that state:

Autoregressive Generation

Autoregressive generation produces output one token at a time, feeding each generated token back as input for the next step.

Backpropagation

Backpropagation computes the gradient of the loss with respect to every parameter in the network by applying the Chain Rule (Multivariable) on the computation graph.

Batch Normalization

Batch normalization normalizes activations within each mini-batch to stabilize and accelerate training.

Bellman Equations

The Bellman equations express the recursive relationship in Value Functions: the value of a state equals the immediate reward plus the discounted value of the next state.

Bias-Variance Tradeoff

The expected test error of a model decomposes into three terms:

Causal Masking

Causal masking restricts Self-Attention so that each position can only attend to itself and earlier positions — never future tokens.

Chain-of-Thought Prompting

Chain-of-thought (CoT) prompting improves LLM reasoning by eliciting intermediate steps before the final answer.

Classical Machine Learning

Classical machine learning studies how models generalize from data without relying on large neural networks. The useful mental model is: choose a hypothesis class, define a loss…

Computational Complexity of Attention

Standard Self-Attention has time and memory in sequence length . This is the fundamental bottleneck of transformers.

Convolutional Neural Networks

CNNs exploit spatial structure through three key ideas: local connectivity, parameter sharing, and translation equivariance.

Cross-Validation

Cross-validation provides honest estimates of model performance on unseen data.

Curse of Dimensionality

As dimensionality increases, data becomes exponentially sparse. This breaks intuitions from low-dimensional spaces and has deep consequences for ML.

Deep Q-Network

DQN extends Q-Learning to high-dimensional state spaces by approximating with a neural network.

Direct Preference Optimization

DPO (Rafailov et al., 2023) trains directly on preference pairs without a separate reward model or RL loop.

Dropout

Dropout randomly sets each neuron's output to zero with probability during training:

Dynamic Programming in RL

Dynamic programming solves MDPs exactly when the model ( , ) is known. Foundation for understanding all RL algorithms.

Embeddings

An embedding maps discrete objects (words, tokens, users, items) into a continuous vector space where geometric relationships encode semantic relationships.

Evaluation Metrics

Classification metrics:

Inductive Bias

An inductive bias is an assumption baked into a model's architecture that constrains what functions it can learn. It encodes prior knowledge about the problem structure.

K-Nearest Neighbors

KNN is a non-parametric algorithm: it stores all training data and classifies new points by majority vote among the nearest neighbors.

KL Penalty

The KL penalty in RLHF constrains the trained policy from drifting too far from the reference (SFT) model.

Layer Normalization

Layer normalization normalizes across the feature dimension for each individual sample:

Linear Regression

Linear regression models the relationship (or with bias absorbed).

LLMs as RL Agents

The framing of LLM generation as a reinforcement learning problem reveals deep connections between language modeling and sequential decision-making.

Logistic Regression

Logistic regression is a linear classifier that models the probability of class membership:

LoRA

LoRA (Low-Rank Adaptation) makes fine-tuning large models practical by training only small low-rank matrices.

Loss Functions

The loss function measures how wrong the model's predictions are. Training minimizes it via Backpropagation.

LSTM and GRU

LSTMs and GRUs solve the vanishing gradient problem of Recurrent Neural Networks through gating mechanisms.

Markov Decision Process

An MDP is the formal framework for sequential decision-making under uncertainty.

Monte Carlo Methods in RL

Monte Carlo (MC) methods learn value functions from complete episodes of experience — no model needed.

Multi-Head Attention

Multi-head attention runs Self-Attention multiple times in parallel, each head learning different relationships.

Multimodality

Multimodal models extend a language model to consume (and sometimes produce) images, audio, or video by mapping each modality into the same embedding space the LLM already…

Neural Networks

Neural networks learn composed functions from data. The useful mental model is: layers apply differentiable transformations, nonlinearities make the composition expressive, and…

Policy Gradient Theorem

The policy gradient theorem gives the gradient of the expected return with respect to policy parameters — enabling direct optimization of the policy.

Positional Encoding

Transformers have no built-in notion of order — Self-Attention is permutation-equivariant. Positional encodings inject position information.

Pretraining

Pretraining teaches an LLM to predict the next token on massive text corpora — the foundational stage of the LLM pipeline.

Principal Component Analysis (PCA)

PCA finds the directions of maximum variance in data and projects onto them for dimensionality reduction.

Proximal Policy Optimization (PPO)

PPO is the default policy gradient algorithm for practical RL, including RLHF.

Q-Learning

Q-learning is an off-policy TD control algorithm that directly learns the optimal action-value function .

Random Forest

Random forest is an ensemble of decision trees that reduces variance through bagging and feature randomization.

ReAct

ReAct (Yao et al., 2023) interleaves reasoning and acting, enabling LLMs to use external tools.

Recurrent Neural Networks

RNNs process sequences by maintaining a hidden state that is updated at each time step:

Regularization

Regularization adds a penalty to the loss function to prevent overfitting by constraining model complexity.

REINFORCE

REINFORCE is the simplest policy gradient algorithm. It uses complete episode returns to estimate the gradient.

Reinforcement Learning

Reinforcement learning studies agents that learn by acting in an environment and receiving reward. The useful mental model is: an MDP defines the interaction, value functions…

Residual Connections

A residual (skip) connection adds the input of a layer to its output:

RLHF Pipeline

RLHF (Reinforcement Learning from Human Feedback) aligns LLMs with human preferences by training against a learned reward model.

Sampling and Decoding Strategies

At each step, an autoregressive language model outputs logits over the vocabulary. The decoding strategy determines how to pick the next token from this distribution.

SARSA

SARSA is an on-policy TD control algorithm. The name comes from the tuple .

Scaling Laws

Scaling laws describe how LLM performance (loss) improves predictably as you increase model size, data, and compute.

Self-Attention

Self-attention is the core mechanism of the transformer. It computes relationships between all pairs of positions in a sequence.

Self-Improvement in LLMs

Self-improvement methods allow LLMs to improve their own capabilities by generating and learning from their own outputs.

Softmax

Softmax converts a vector of raw scores (logits) into a probability distribution:

Supervised Fine-Tuning

SFT fine-tunes a pretrained LLM on instruction-response pairs, teaching it to be a helpful assistant instead of a text completer.

Temporal Difference Learning

TD learning combines the model-free nature of Monte Carlo with the bootstrapping of dynamic programming.

Test-Time Compute

Test-time compute refers to strategies that spend more computation at inference to improve output quality, rather than scaling the model itself.

The Artificial Neuron

The fundamental unit of a neural network:

Tokenization

Tokenization converts raw text into the integer sequences that transformers process.

Transformers and LLMs

Transformers and LLMs model token sequences by repeatedly mixing contextual information across positions. The useful mental model is: tokenization creates discrete symbols…

Value Functions

Value functions estimate how good it is to be in a state (or to take an action in a state) under a given policy’

Weight Initialization

Wrong initialization causes vanishing or exploding gradients before training even begins.