A pattern I've seen repeat across more launches than I'd like to admit: a feature ships with two or three plausible reasons it might help, nobody picks one of them as the goal metric ahead of time, and once the results come in, the readout quietly becomes a search for whichever one of those reasons the data happens to support. It never feels like cheating while it's happening. It is, statistically, almost exactly that.
How It Actually Plays Out
The shape is always similar. Going into the launch, there are two or three genuine hypotheses for why the change should be good: maybe it deepens sessions, maybe it lifts conversion, maybe it reduces churn a few weeks out. All three are reasonable. Nobody wants to throw any of them away before looking, because any one of them could turn out to be the real story.
Then the results come in, and the process becomes: check the metric for hypothesis one. Not significant. Check the metric for hypothesis two. Not significant. Check the metric for hypothesis three — and there it is, a clean p-value under 0.05. That becomes the headline of the readout. The other two hypotheses quietly disappear from the conversation, not because they were wrong, but because they didn't happen to clear the bar this time.
Why It Doesn't Feel Like a Problem in the Moment
Nobody doing this thinks of it as p-hacking, and that's precisely what makes it durable. Each hypothesis was genuinely plausible before anyone looked at data — this isn't a case of inventing a story after the fact to fit noise. Checking all of them feels like thoroughness, like not leaving value on the table by only looking in one place. The problem isn't that any individual check is dishonest. The problem is what happens statistically once you check several and stop as soon as one clears the bar.
The Math That's Actually Running in the Background
Treat the three hypotheses as three roughly independent tests, each run at the standard . The probability that at least one of the three clears significance purely by chance, even if none of the underlying effects are real, is:
Not 5%. Closer to 14.3% — nearly three times the error rate anyone believes they're operating at, because the decision rule was actually "stop at the first hypothesis that passes," not "test one pre-specified hypothesis." Every hypothesis added to the list before launch is another independent roll of the dice at decision time, whether or not anyone frames it that way.
Why This Is Worse Than a One-Time Inflated Error Rate
The 14.3% number describes one launch. The habit is worse than the number, because it's self-reinforcing: once a team learns that having three plausible hypotheses instead of one gives a meaningfully better chance of finding something to celebrate, the natural response is to write more hypotheses before every launch, not fewer. Nobody has to act in bad faith for this to happen — every hypothesis on the list looked reasonable when it was written down, and the search only ever stops at the first one that passes, never at the ones that don't. Across many launches, this quietly pushes the effective false-positive rate for "the org believes this shipped a real win" well above what anyone thinks they've agreed to.
It's worth naming the family resemblance here: this is the same shape as the peeking problem — checking a result repeatedly and stopping the moment it looks good — just running along a different axis. Peeking compounds false positives across time. This compounds false positives across hypotheses. Neither one requires anyone to act dishonestly for the error rate to blow past what the team believes it's protecting.
The Fix: Pick the Goal Metric Before You Look
The fix has to happen before data collection starts, not during the readout. Name one metric as the goal metric — the single thing that decides ship or no-ship — and write it down before anyone has seen a result. The other hypotheses don't have to be discarded; they can still be checked afterward for learning. What changes is their authority: only the pre-registered goal metric gets to decide the launch. Everything else is exploratory, worth a look, and explicitly not allowed to override the metric that was actually agreed to in advance.
What Happens When the Goal Metric Doesn't Pass
This is the part that requires actual discipline, because it's the part with a real cost: if the pre-registered goal metric comes back non-significant, the honest move is to roll back — not to go looking through the other hypotheses for one that happened to move. A non-significant goal metric that gets rescued by a different metric found after the fact hasn't been rescued. It's just been replaced by the result of the exact search process described above.
And if, on top of a non-significant goal metric, other usability-related signals have also declined — the kind of local-placement regression covered in the previous retrospective in this series — that combination should make the case for rolling back stronger, not weaker. A goal metric that didn't clear the bar is a statistical signal. A usability metric that declined at the same time is a mechanism you can point to. Together, they stop being "the test was inconclusive" and start being "there's a reason this made the platform worse, and the number that was supposed to prove it helped never came through either." That's a materially stronger basis for a rollback than a lone non-significant p-value, and a materially weaker basis for talking yourself into a different metric instead.
The Thread Connecting Both Retrospectives
Both of these lessons come from the same root cause: not deciding, in advance, what "success" actually means and what evidence would change your mind. Without a predefined goal metric, a team is exposed to picking whichever interpretation of the data supports the launch it already wanted to ship — the same vulnerability that made the rollout call in the previous article genuinely hard to resolve cleanly. The fix for both is the same discipline, applied earlier than most teams are used to applying it: decide what you're actually trying to prove, and what would make you walk away, before you have any data available to talk yourself out of it.