In a fixed-horizon A/B test, you calculate the sample size in advance, wait until every observation reaches that planned point, and check the result exactly once. In practice, almost nobody has that much patience. Engineers want to know if the feature is working. Product managers want to ship faster. Everyone wants the experiment to clear the bar quickly. The temptation to peek at results mid-experiment is universal.
But this kind of peeking is statistically dangerous — it raises the probability of wrongly concluding there's an effect when none actually exists.
The Standard A/B Test Assumes You Only Look Once
Setting a significance level of means the following: if there is truly no effect, the probability of wrongly claiming the treatment differs from control is only 5%. But this guarantee only holds if you look at the data exactly once, at the end. In other words, it caps the probability of a false positive — wrongly concluding there's an effect when there isn't one — at 5%.
If you check results multiple times along the way — especially if you stop early because things look good — that pre-set significance level statistically falls apart. Checking repeatedly and only stopping early on good-looking results means every additional look creates one more chance for a false positive by pure chance.
Why Peeking Inflates Your False Positive Rate
A Simple Example
Suppose you run a four-week experiment and check the p-value at the end of each week. Even if the treatment has no real effect at all, the probability that at least one of those four checks turns up falsely significant is:
You think you're running the experiment at a 5% false positive rate. You're actually running it at 18.5%. The more often you check, the worse this gets — with 20 looks, it climbs past 64%.
What Sequential Testing Does Differently
Sequential A/B testing doesn't forbid you from looking at results early. Instead, it adjusts the threshold you need to clear at each checkpoint, so the overall false positive rate stays controlled the way you intended.
The core idea: your total error budget is . Instead of spending all of it on one final check, you distribute it across several planned checkpoints. Each checkpoint is allocated a share of the budget, and its significance threshold is calibrated to match that share exactly.
- Early checks get a very small share of the budget — so the threshold is very high. You need strong evidence to stop early.
- Later checks get more of the budget — so the threshold moves closer to the familiar 1.96.
- Add up the alpha spent across every check, and the overall false positive rate stays at exactly 5%.
The Key Constraint: Pre-Registration
Sequential testing works in practice only when the check schedule is decided before the experiment starts. You commit in advance to how many times you'll check, and at which points in the data-collection process.
The error-control guarantee depends entirely on this. If you add an extra check because the results look interesting, or change the stopping rule mid-experiment, the false-positive-rate guarantee no longer holds.
When Sequential Testing Matters Most
Not every experiment needs sequential testing. If you can commit to a fixed sample size and have no operational pressure to stop early, a standard fixed-horizon test is simpler and equally valid.
Sequential testing earns its complexity in the following cases:
- You genuinely need to stop early. Safety monitoring, high-cost experiments, or a fast-moving product cycle where waiting for the full sample is operationally impractical.
- Stakeholders are going to peek regardless. If results are going to get checked mid-experiment no matter what, sequential testing at least makes the error implications explicit and controlled.
- Sample size is hard to fix in advance. When traffic or conversion rates are uncertain, sequential methods offer flexibility without sacrificing statistical rigor.
What Comes Next
How to distribute the error budget across multiple checks — and how to set the threshold at each one — is handled by alpha spending functions, covered in the next article. The concept covered here is the theoretical foundation for that article.
Sequential testing isn't about ignoring statistical discipline — it's a rule that extends that discipline to the way experiments actually get run in practice.