How Many Guardrails Is Too Many? The Other Multiplicity Problem

The previous article showed that a single non-inferiority test on a single guardrail already needs more data than most teams provision. Almost no real experiment stops at one guardrail, though — a typical launch decision watches revenue, latency, and one or two engagement or complaint metrics at the same time. Running several guardrail tests at once introduces a second multiplicity problem, and it isn't the one most teams already know to watch for.

The Familiar Multiplicity Problem, Briefly

An earlier article in this blog covered the well-known version of multiple testing: run 20 independent significance tests and the chance that at least one shows a false positive by pure chance climbs to roughly 64%, even if nothing real is happening anywhere. That's a problem on the $\alpha$ side — false positives compounding as you add more tests. The standard fixes (Bonferroni, Benjamini–Hochberg) are built to keep that compounding under control.

Guardrails, run under a non-inferiority framework, compound in a different place entirely.

The Guardrail Decision Rule

The sensible way to use $G$ guardrails in a launch decision is conjunctive: ship only if every one of the $G$ guardrails individually clears its own non-inferiority test. One failing guardrail is enough to block the launch, or at least trigger a closer look. This "all must pass" rule is standard practice, and for good reason — it's also, reassuringly, not the source of inflated false positives. Requiring every test in the family to reject its own "worse than $\delta$" null actually keeps the family-wise Type I error at or below each individual test's $\alpha$, no correction needed. The compounding problem here isn't about false positives at all.

Where the Real Cost Shows Up: Power

The conjunctive rule is safe on the false-positive side, but it's expensive on the other side. Suppose the treatment truly is non-inferior on every one of your $G$ guardrails, and each individual test is designed with power $1-\beta$ to correctly detect that. The probability that all $G$ independent tests correctly pass — the probability your launch decision doesn't get needlessly blocked by chance — is:

This shrinks fast. A guardrail suite where every individual test is well-powered at 90% still only has a combined chance of sailing through cleanly that falls well below 90% once you're checking more than one or two metrics:

With five guardrails, each individually sized to a respectable 90% power, the family as a whole only has a 59% chance of passing cleanly when the treatment genuinely is safe everywhere. The other 41% of the time, a perfectly good launch gets needlessly stalled because at least one truly-fine guardrail randomly missed its own target — not because anything is actually wrong.

Correcting for It

The fix mirrors Bonferroni, but applied to $\beta$ instead of $\alpha$. If you want the entire launch decision — the goal metric's result plus every one of the $G$ guardrails — to hit a target family-wise power of $1-\beta_{\text{target}}$, split the tolerable miss-rate evenly across all $G+1$ components of that decision (the $+1$ accounts for the goal metric's own result, which has to come through correctly too for the launch to happen as intended):

For a target family-wise power of 80% ($\beta_{\text{target}} = 0.2$), here's what that means for the power each individual test needs to be designed to:

Why divide, not just accept the erosion: the $\beta/(G+1)$ split is a simple, conservative first pass — it assumes independence across guardrails, which is rarely exactly true (revenue and average order value, for instance, tend to move together). Just like the correlation caveat raised in the alpha-spending worked example earlier in this series, treating guardrails as independent is a reasonable back-of-envelope approximation, not an exact answer — the exact correction depends on how correlated your specific set of guardrails actually is.

This Compounds With the Margin Problem

The previous article showed that required sample size scales with $(z_\alpha + z_\beta)^2$, and that shrinking the margin $\delta$ already inflates that requirement quadratically. Raising the target power per test compounds it further, because $z_\beta$ itself grows with power: at 80% power, $z_\beta \approx 0.842$; at 95% power, $z_\beta \approx 1.645$. Holding everything else fixed, moving from 80% to 95% per-test power alone increases the required sample size by about 1.75×. Add three or four correlated guardrails, each with a tight margin, each now needing 95%+ individual power to hold the family together — and the total sample requirement for a "simple" guardrail suite can end up many times larger than the goal metric's own sample size calculation ever accounted for.

What Comes Next

Two open questions remain, and they're the two hardest, most practical parts of this whole series. First: given all of this machinery, what should the margin $\delta$ actually be set to for a real metric like revenue? That's a business decision as much as a statistical one, and it's worth a full worked example on its own. Second: once you have real margins, correlated guardrails, and a corrected power target, what sample size do you actually need to plan for? The next two articles in this series answer both, in that order.