Modern experimentation platforms rarely test a single metric. Recommendation systems track click-through rate, watch time, retention, and revenue simultaneously. Advertising experiments segment results across countries, devices, creatives, and user cohorts. The statistical problem is that every additional test increases the probability of false discoveries.
If you ignore multiple testing entirely, your experiment platform eventually becomes a machine for generating fake wins. Understanding why this happens — and what your options are — is the foundation before reaching for any correction method.
Why Multiple Testing Becomes Dangerous
Family-Wise Error Rate Explodes
Suppose you test 20 independent metrics with significance level . Even if all null hypotheses are true, the probability of observing at least one false positive is:
That means there's roughly a 64% chance of reporting at least one statistically significant result purely by noise. The more tests you run, the more this probability climbs toward 1.
This isn't a flaw in any specific test — it's a fundamental property of probability. Each test carries its own false positive risk, and those risks accumulate across the experiment.
Where This Happens in Practice
- Recommendation systems evaluating dozens of engagement metrics simultaneously
- Ads experiments sliced by geography, platform, and campaign type
- Product teams monitoring many secondary outcomes alongside a primary metric
- Data scientists exploring multiple model variants in the same experiment
Without any correction, "significant" results in these settings become increasingly unreliable. The platform starts rewarding noise over signal.
Two Different Ways to Think About Error
Before choosing a correction method, you need to decide what kind of error you're trying to control. There are two fundamentally different objectives:
- Family-Wise Error Rate (FWER): the probability of making even one false positive across all your tests. Controlling FWER means you want to be nearly certain that every significant result is real.
- False Discovery Rate (FDR): the expected proportion of false positives among all results you declare significant. Controlling FDR means you're willing to accept some false discoveries, as long as they stay below a defined fraction.
These objectives lead to very different behaviors in practice. FWER control is conservative — it protects aggressively but often misses real effects. FDR control accepts that large-scale discovery is inherently noisy, and manages that noise rather than trying to eliminate it entirely.
The Core Tension
Every multiple testing correction involves a tradeoff between two things:
- Type I error control — avoiding false positives
- Statistical power — detecting real effects when they exist
Tighter error control always costs power. As you correct more aggressively, smaller real effects become harder to detect. In large-scale experimentation, this tradeoff becomes operationally significant: an overly conservative system slows down learning and misses genuine improvements.
What Comes Next
Once you understand the structure of the problem, the two most widely used corrections are Bonferroni (which controls FWER) and Benjamini–Hochberg (which controls FDR). They solve different problems and suit different experimental contexts.
In recommendation systems and advertising experiments, where hundreds of hypotheses may be tested simultaneously, FDR control is usually the more practical default. Bonferroni is better suited for high-stakes confirmatory decisions where even a single false positive is unacceptable.
The key takeaway from this article: statistical significance does not exist in isolation. The moment you test multiple hypotheses simultaneously, error control becomes a system-level problem — not a single-test problem.