Every team wants their experiment to win. That single fact explains almost everything wrong with how A/B tests get monitored in practice. Nobody sits down, calculates a sample size, waits patiently, and checks the result exactly once. Instead, the dashboard gets refreshed every morning, every afternoon, sometimes every hour — and the experiment quietly stops the moment it first looks good. This is the peeking problem, and it is one of the most common ways experimentation programs generate false wins without anyone realizing it.
The Behavior: Refresh Until It Looks Good
Picture a typical experiment. A PM ships a new checkout flow. The team is excited. Someone opens the experiment dashboard the next morning to check conversion rate. It's not significant yet — fine, too early. They check again the next day. Still not quite there. By day four, the p-value dips under 0.05. Champagne emojis in Slack. The experiment is called a win and shipped to 100% of traffic.
Nothing about this looks reckless. Everyone involved would say they're being careful — they didn't stop on day one, they waited for "real" significance. But the behavior itself is exactly what statisticians call optional stopping: checking the result repeatedly and stopping as soon as it crosses a threshold. And it has a very specific, well-understood consequence.
Why Refreshing the Dashboard Inflates False Positives
A significance threshold of means something very precise: if there is truly no effect, there is a 5% chance a single test will falsely show significance. That guarantee is only valid if you look at the data exactly once.
The moment you check repeatedly and stop the first time the result looks favorable, you are no longer running one test — you are running many, and each additional look is another chance for random noise to cross the line. Suppose you check once a day for four days. Even if the new checkout flow does absolutely nothing, the probability that at least one of those four checks shows a "significant" result by pure chance is:
You believed you were running at a 5% false-positive rate. You were actually running close to 18.5%. Teams that check daily for two or three weeks — which is extremely common — push this number well past 50%. Almost any experiment, given enough looks, will eventually appear to "win" on noise alone.
The Guardrail Version Is Just as Dangerous
Most teams have learned, at least informally, to also watch a guardrail metric — revenue, latency, unsubscribe rate — to make sure the primary win isn't hiding a regression somewhere else. This sounds like the responsible thing to do. In practice, it usually just adds a second round of the same problem.
Here's the pattern: the team keeps refreshing the dashboard until the primary metric looks good and the guardrail doesn't look obviously bad. But "doesn't look obviously bad" is not a statistical control — it's a vibe check performed on noisy data, repeated until it passes. Guardrail metrics fluctuate day to day just like primary metrics do. If you keep checking until the guardrail happens to look fine on a good day, you haven't verified safety — you've simply found the one snapshot, out of many noisy snapshots, that let you stop.
This creates two compounding failure modes at once:
- False wins: the primary metric crosses significance purely from repeated looks, not a real effect.
- False safety: the guardrail metric happens to look acceptable on the day you decided to stop, even if the treatment is quietly causing harm on average.
Neither of these requires bad intent. They are the predictable result of checking repeatedly without a plan for how many looks you're allowed and what threshold each one has to clear.
Why This Keeps Happening
Peeking isn't a discipline problem you can fix by asking people to "just wait longer." It happens because the incentives are real: engineers want to know if their work paid off, PMs are measured on shipped impact, and waiting three weeks for a single look feels wasteful when the dashboard updates in real time. Refreshing is the path of least resistance, and nothing about the tooling stops anyone from doing it.
The honest fix isn't "stop looking." It's designing a monitoring plan that expects people to look — and controls the error rate anyway.
What Comes Next
A common first response to the peeking problem is to pick a fixed check-in day — say, seven days in — and refuse to make a decision before then. It sounds like a reasonable compromise. The next article, why a fixed 7-day rule doesn't actually solve peeking, walks through exactly where that compromise breaks down — and why guardrail metrics are still exposed even when the primary metric is protected.