After getting burned by the peeking problem, a lot of teams land on the same fix: pick a fixed number of days — seven is the classic choice — and refuse to make a call before then. It's a sensible instinct. It also only solves half the problem, and the half it leaves unsolved is usually the one that matters more.
What the Fixed Window Actually Fixes
If a team truly commits to a single decision point — no informal peeking, no early calls, one look at day 7 — the fixed-horizon test is statistically valid. The 5% false positive rate holds exactly, because you're back to the textbook assumption: one test, one look. This is the entire appeal of the rule. It's simple, it's easy to explain to stakeholders, and it directly targets the optional-stopping behavior described in the previous article.
Where It Breaks Down
The Discipline Is Usually Fake
In practice, "we don't look before day 7" rarely means nobody looks before day 7. It means nobody makes it official before day 7. The dashboard still gets checked daily — informally, "just to see how it's trending." When day 7 arrives and the metric happens to be favorable, the team calls it a win using the fixed-window rule as cover. When day 7 arrives and the metric looks unfavorable, the same team often extends the test "to be safe" or waits for "more signal." The rule only gets enforced when it's convenient. That's optional stopping with extra steps.
The Guardrail Metric Gets One Noisy Snapshot
This is the bigger issue, and it's the one that fixed windows don't even attempt to solve. The primary metric gets one protected look at day 7. But guardrail metrics — revenue, latency, complaint rate — also only get checked once, at that same single point in time, with no correction and no ability to detect a problem before then.
A guardrail metric is just as noisy as a primary metric. If revenue happens to be down 2% on day 7 purely from random daily fluctuation — not because the treatment caused harm — the team has no way to tell the difference between "the treatment hurt revenue" and "revenue was just having a bad day." There's no sequence of looks to show whether the dip is a trend or a blip. One snapshot can't distinguish signal from noise, and a single arbitrary checkpoint gives you exactly one snapshot.
Worse: if the treatment is causing real, meaningful harm on day 2, the fixed-window rule says wait anyway. There's no mechanism to stop early for safety, because the whole point of the rule was to prevent any decision before day 7 — for better or worse.
The Number Itself Is Arbitrary
Why seven days? Usually because it covers one full weekly cycle, which is a reasonable practical reason — but it isn't derived from anything about how fast the true effect resolves, how much traffic the experiment gets, or how costly a wrong decision would be. It's a round number that feels safe. Two different experiments with very different risk profiles and different traffic volumes often get the exact same "day 7" rule, simply because that's the convention.
The Real Trade-Off
A strictly-enforced fixed window is not wrong — it correctly controls the primary metric's false positive rate, which is a real improvement over uncontrolled peeking. But it buys that control at a steep price:
- No legitimate way to stop early for a clear win, even with strong evidence.
- No legitimate way to stop early for clear harm — the guardrail is checked once, blind to everything that happened before that point.
- No statistical difference between day 7 being genuinely representative and day 7 just being a noisy day.
In other words: the fixed window solves peeking by removing the ability to look at all — which also removes the ability to react to real signal, in either direction, before the deadline.
What You Actually Want
The right solution isn't "look once" or "look whenever you feel like it." It's looking a small, pre-planned number of times — say at day 1, day 3, and day 7 — with each look assigned its own statistically calibrated threshold, so that checking early doesn't inflate your overall false positive rate and doesn't leave guardrail metrics unmonitored until the very end. That framework is called Group Sequential Testing, and it's the subject of the next article.