Statistical Significance vs Practical Significance

1 min read

# topic/math/statistics

Statistical significance asks whether an observed effect is unlikely under a null hypothesis.

Practical significance asks whether the effect is large enough to matter.

With enough data, tiny effects can become statistically significant:

small effect+huge n→small p-value\text{small effect} + \text{huge } n \to \text{small p-value}

With too little data, useful effects can fail to reach statistical significance:

large effect+small n→uncertain estimate\text{large effect} + \text{small } n \to \text{uncertain estimate}

Effect size matters:

  • Difference in means
  • Relative improvement
  • Odds ratio
  • Cohen's dd
  • Absolute metric gain in ML

Example in ML: a model improves accuracy from 94.1% to 94.2%. This might be statistically significant on a huge test set, but not worth extra latency, cost, complexity, or risk.

Decision quality requires both:

  • Estimate the effect
  • Estimate uncertainty
  • Compare against the cost of acting

Rule: do not report only p-values. Report effect size and uncertainty.

See also: Hypothesis Testing, Confidence Intervals, Evaluation Metrics

Linked from