Every experiment carries two different kinds of metrics, even when nobody labels them that way. There's the metric you're trying to move — sign-ups, checkout conversion, watch time — and there are the metrics you're trying to protect while you move it — revenue, latency, unsubscribe rate. Almost every experimentation platform runs both through the exact same statistical test. That's the mistake this article is about.
Two Metrics, Two Different Questions
Call the first kind a goal metric: the thing the experiment exists to improve. The question you're asking of a goal metric is "did this get better?" You want evidence of a difference, specifically in a good direction, and you're happy to conclude nothing happened if the data doesn't clearly show an improvement.
Call the second kind a guardrail metric: the thing the experiment must not quietly break. The question you're asking of a guardrail is completely different — not "did this change?" but "did this stay okay?" You're not hoping to detect a difference. You're hoping to rule one out.
Those are not the same question, and treating them as if they were is where most experimentation programs quietly go wrong. A test built to answer "did this change?" does not automatically answer "did this stay okay?" — even though it's tempting to assume it does, because it's the only test most teams have in the toolbox.
The Test Everyone Reaches For
The standard two-sample test used for goal metrics sets up a null hypothesis of "no difference" and an alternative of "there is a difference":
This is a superiority test. It's built to detect a difference, and it is very good at that job. When the p-value crosses your threshold, you have real evidence that treatment and control aren't the same. That's exactly what you want when you're evaluating a goal metric.
The trouble starts when the same test gets pointed at a guardrail metric — revenue, say — with the same null hypothesis of "no difference." A non-significant p-value on that test tells you that you failed to find evidence of a difference. It does not tell you that no difference exists. Those are not the same claim, and the gap between them is exactly where guardrail failures hide.
Why "Failed to Reject" Isn't "Proved Safe"
A p-value answers one specific question: assuming there is truly no effect, how surprising is data this extreme? If the data isn't surprising enough, you fail to reject the null — but "not surprising enough to reject" covers an enormous range of realities. It covers the case where the treatment truly does nothing. It also covers the case where the treatment causes real harm, but your sample was too small or too noisy to detect it. The test can't tell those two situations apart, because it was never designed to distinguish them. It was designed to catch differences, not to certify their absence.
This is the textbook distinction between "absence of evidence" and "evidence of absence." A superiority test gives you the first. Confirming a guardrail metric is safe requires the second — and the second requires a fundamentally different test construction, which is the subject of the next two articles in this series.
A Concrete Version of the Problem
Suppose a checkout redesign is being tested. The goal metric — conversion rate — was carefully powered: the team ran a sample size calculation, decided they needed to detect a 2 percentage point lift, and collected enough traffic to do that reliably.
Revenue per user rides along as a guardrail on the same sample. Nobody ran a separate power calculation for it — why would they, it's "just being monitored." At the end of the experiment, revenue per user is up 0.4% with a p-value of 0.32. Not significant. The team reads this as "no meaningful impact on revenue" and ships.
But revenue is a noisier, higher-variance metric than conversion rate, and the sample was sized for conversion — not for revenue. The 95% confidence interval around that 0.4% point estimate might easily run from to . A non-significant p-value on an interval that wide isn't reassurance. It's a wide net that happened to also catch zero — along with a lot of other numbers, including some genuinely bad ones.
Why Guardrails Are Systematically Underpowered
This isn't a one-off mistake — it's the default outcome of how experiments get designed. Sample size is almost always calculated for the goal metric alone, using the minimum detectable effect the team actually cares about for that metric. Guardrail metrics are added afterward, evaluated on whatever sample the goal metric happened to need, with no separate power analysis of their own.
- Guardrails are often higher-variance than the goal metric — revenue and latency both tend to have fatter tails than a binary conversion event, which inflates the standard error for a given sample size.
- Guardrails are rarely powered on purpose. Nobody asks "how small a revenue drop do we need to be able to detect?" before launch — the guardrail just inherits whatever sample the goal metric was sized for.
- A wide, underpowered interval will contain zero almost by default, regardless of whether a real (and meaningful) regression is happening underneath it.
Put those three together, and "non-significant guardrail" stops meaning "guardrail confirmed safe" and starts meaning "we didn't collect enough evidence to know either way" — which is a very different thing to write in a launch decision doc.
What a Guardrail Test Actually Needs
If a superiority test can't answer "did this stay okay?", something else has to. That something else needs to flip the null hypothesis around: instead of assuming "no difference" and looking for evidence against it, it should assume "the treatment is meaningfully worse" and require the data to actively rule that out. Only rejecting that null gives you the kind of evidence a guardrail decision actually needs — a positive confirmation of safety, not a failure to find harm.
That flipped-null approach has a name — non-inferiority testing — and it's built specifically to answer the guardrail question instead of the goal-metric question. The next article in this series looks at what happens in practice when teams skip this distinction: a "flat" guardrail metric that gets read as safe, when it was never powered to tell the difference between safe and quietly broken.
What Comes Next
Before getting into the mechanics of non-inferiority testing, it's worth sitting with just how common — and how invisible — the "flat means fine" mistake is. The next article walks through exactly how a genuinely harmful change survives a guardrail check and ships anyway, using nothing more exotic than a small sample and an honest reading of a p-value.