Why Non-Inferiority Tests Need Bigger Samples (and How Much)

The previous article defined non-inferiority testing: flip the null hypothesis, require the data to rule out harm bigger than a pre-agreed margin $\delta$. That fixes the logic problem. It also quietly creates a practical one — ruling out a bounded range of harm is a harder statistical task than just detecting whether any difference exists at all, and it usually takes more data than teams expect to bring to a guardrail metric.

The Superiority Baseline

Start from the sample size formula most teams already use for a goal metric. To detect a minimum effect $\Delta$ with power $1-\beta$ at significance $\alpha$ (two-sided), the required sample size per arm is approximately:

Everything here is familiar: bigger variance $\sigma^2$ needs more data, a smaller effect you're trying to detect needs more data (the $\Delta^2$ in the denominator), and more power or a stricter alpha needs more data too.

The Non-Inferiority Version

A one-sided non-inferiority test, assuming the treatment and control are truly equal on average (the standard conservative planning assumption), swaps in the margin $\delta$ for the effect size, and uses a one-sided critical value:

Structurally, this looks almost identical to the superiority formula — same shape, same variance term, same power term. The two changes are easy to miss and pull in opposite directions: the critical value moves from $z_{\alpha/2}$ to the slightly smaller one-sided $z_{\alpha}$ (which shrinks the required sample a little), but the effect size in the denominator moves from $\Delta$ to $\delta$ — and in practice, that single substitution is what actually determines whether you need more data or less.

Why the Substitution Usually Hurts

The one-sided critical value change is small: $z_{0.025} = 1.96$ versus $z_{0.05} = 1.645$ is only about a 15% reduction in that term. Combined with the same power requirement in both formulas, that difference alone brings required sample size down by roughly 20% when the margin equals the effect size. That would make non-inferiority tests slightly cheaper, if the effect size in the denominator stayed the same.

It doesn't stay the same, and this is the part that catches teams off guard. When a metric is a goal metric, the minimum detectable effect $\Delta$ is chosen to be "the smallest improvement worth having" — often a fairly generous number, because small improvements aren't very interesting anyway. When the same metric is a guardrail, the margin $\delta$ is chosen to be "the largest harm we'll tolerate" — and for a metric like revenue or retention, teams reasonably want that number to be small. A small $\delta$ in the denominator, squared, dominates the modest saving from the one-sided critical value every time.

Putting Numbers on It

Holding $\sigma$, $\alpha = 0.05$, and 80% power fixed, and comparing required sample size against a reference superiority test with $\Delta = 2\%$, here's how the non-inferiority sample size scales as the margin $\delta$ shrinks relative to that same 2% reference:

The pattern is quadratic, not linear: halving the margin roughly quadruples the sample size, because $\delta$ is squared in the denominator. A team that wants tight, meaningful protection on a guardrail — a margin noticeably smaller than the effect size they'd have used for a goal metric — should expect to need several times the sample, sometimes an order of magnitude more, not the same sample the experiment was already collecting for its primary metric.

Why This Explains the Silent Failure From Before

This is the quantitative version of the problem raised two articles ago: guardrails inherit whatever sample the goal metric happened to need, with no separate sizing of their own. Now it's clear exactly why that inheritance falls short. A goal metric's sample size is calibrated to detect its own (often generous) minimum effect. A guardrail that wants a tight margin needs several times more data than that — the two requirements are rarely the same number, and the guardrail's is usually larger.

What Comes Next

This math already tells you the mechanical cost of a given margin. It doesn't tell you what the margin should actually be — that's a business decision, not a formula, and it's substantial enough to deserve its own worked example later in this series. Before that, there's a second sample size problem hiding just underneath this one: most experiments don't watch one guardrail, they watch several at once. The next article covers what happens to your error budget — and your sample size — once you're running more than one non-inferiority test in the same experiment.