Random Forest

Random forest is an ensemble of decision trees that reduces variance through bagging and feature randomization.

Algorithm:

1. For each tree $t = 1, \dots, T$:
   • Draw a bootstrap sample (random sample with replacement) from training data
   • Grow a decision tree, but at each split consider only a random subset of $\sqrt{d}$ features (classification) or $d/3$ features (regression)
2. Aggregate: majority vote (classification) or average (regression)

Why it works:

• Individual trees are high-variance, low-bias estimators
• Bagging (bootstrap aggregating) reduces variance by averaging many trees
• Feature randomization decorrelates the trees → further variance reduction
• Overall: low bias + low variance (the best of both sides of the Bias-Variance Tradeoff)

Advantages:

• Works well out-of-the-box with minimal hyperparameter tuning
• Handles nonlinear relationships, interactions, mixed feature types
• Feature importance: measure how much each feature improves splits across all trees
• Robust to outliers and noise

Limitations:

• Not interpretable (unlike a single tree)
• Slow prediction for very large forests
• Does not extrapolate well beyond training range (tree-based models are piecewise constant)

See also: Bias-Variance Tradeoff, Cross-Validation, Evaluation Metrics