Two Kinds of Guardrail Tests, Two Different Corrections (and Where I Mixed Them Up)

While reviewing a guardrail readout, I caught myself stuck on a question that should have had an immediate answer and didn't: a metric hadn't gotten worse — if anything it looked good — but there was a real chance I'd end up calling it bad anyway. Was that power? It took working through it on paper to realize I'd been quietly running two different tests together in my head, and they don't take the same correction.

The Muddled Starting Point

The scribbled version of the question, as I actually wrote it down at the time: a non-inferiority test is supposed to prove "not bad." So what does power even mean here? And separately — if the treatment wasn't bad, was actually good, but the test still judged it as bad, is that the thing power is supposed to protect against? I wasn't sure, and the fact that I wasn't sure was the actual problem worth solving, not the specific guardrail in front of me.

Getting the Direction Right

The confusion cleared up once I separated two genuinely different tests that both get casually called "testing the guardrail."

• A standard significance test asks "did this get worse?" — null hypothesis of no difference, and you're looking for evidence against it. This is the familiar test everyone learns first.
• A non-inferiority test asks the opposite question — covered in an earlier article on this blog — flipping the null to assume the treatment is meaningfully worse (by more than a pre-agreed margin $\delta$), and requiring the data to actively rule that out.

Under the non-inferiority framing, my scribbled question actually has a clean answer. Power is the probability of correctly concluding "not bad" when the treatment truly is not bad. The mistake I was describing — "wasn't bad, was actually good, but got judged as bad" — is precisely the failure that reduces power: a real miss, where a genuinely fine (or better) treatment fails to clear the non-inferiority bar anyway, usually from too little data relative to how tight the margin is. My instinct in the moment was actually correct. I just hadn't said out loud which of the two tests I was even asking the question about, which is exactly why it felt unresolved.

Why the Correction Direction Flips

The part that actually tripped me up came next: how many guardrails you're watching changes things, but not in the same way for these two tests.

Run the standard significance test on each of $G$ guardrails, flagging concern if any one of them shows a significant negative move, and the risk that compounds is a false alarm — the classic multiplicity problem covered earlier on this blog: even with no real harm anywhere, the chance that at least one of $G$ independent tests turns up significant by pure noise grows with $G$. The fix here is the familiar one — Bonferroni-style correction of $\alpha$, or FDR control — and it lives entirely on the false-positive side.

Run non-inferiority tests instead, requiring all $G$ guardrails to individually clear their own margin before you'll call the launch safe, and the risk that compounds is the opposite kind of mistake — not a false alarm, but a false all-clear failing to materialize even when the treatment is genuinely fine everywhere:

This is exactly the correction worked through in an earlier article — splitting the target miss-rate across all $G+1$ parts of the launch decision, which means raising each individual test's power, not tightening its significance threshold. It lives entirely on the power side, and it's really a correction in the opposite direction from Bonferroni: one shrinks a threshold to avoid false alarms, the other raises a power target to avoid false all-clears.

Why I Conflated Them in the First Place

Both of these get informally described as "the guardrail multiplicity problem," which makes them sound like the same correction applied to a slightly different setting. They aren't. They answer different questions (did this get worse, versus can we rule out meaningful harm), they compound in different directions (more false positives, versus less family-wise power), and they call for opposite kinds of adjustment (a stricter threshold, versus a higher power target). As covered earlier, a standard significance test was never built to prove safety in the first place — which is a good reason most mature guardrail programs should be running the non-inferiority version, not the significance-test version, as their actual ship gate. But "should be" isn't "obviously is," and it's entirely possible to reason about a guardrail suite while unconsciously borrowing the wrong one's intuition — which is exactly what had me stuck on a question that, once named correctly, wasn't actually hard.

The Practical Habit This Left Me With

Before reaching for a correction, name the decision rule out loud first: is a guardrail "failing" defined as "a standard test found significant harm," or as "the treatment couldn't be confirmed to stay within an acceptable margin"? The first calls for Bonferroni or FDR. The second calls for the $\beta/(G+1)$-style power correction. Getting that one identification right up front would have saved me the confusion — the math on either side isn't the hard part; knowing which math applies is.