Experimental design is about collecting data so comparisons support valid causal or statistical conclusions.
Bad design cannot usually be fixed by a better model.
Core ideas:
- Treatment: intervention or condition being tested
- Outcome: measured response
- Control group: baseline comparison
- Randomization: assigns units to conditions without systematic bias
- Blinding: hides treatment assignment to reduce measurement or behavior bias
Confounding: a variable affects both the treatment and the outcome, creating a misleading association.
Example: if people who choose a training course are already more motivated, then higher post-course performance may reflect motivation, not the course.
Randomization breaks confounding in expectation by making treatment assignment independent of pre-treatment variables.
Common threats:
- Selection bias
- Survivorship bias
- Data leakage
- Multiple testing
- Distribution shift
- Post-treatment adjustment
In ML:
- Train/test leakage inflates evaluation
- Tuning on the test set makes the test set part of training
- Benchmark contamination makes model evaluation unreliable
- A/B tests need random assignment and enough power
Rule: before modeling, ask whether the data collection process supports the claim you want to make.
See also: Cross-Validation, Hypothesis Testing, Statistical Power