Sequential A/B Testing and Alpha Spending

Running an experiment without a fixed sample size and peeking at results along the way is one of the most common — and most dangerous — practices in A/B testing. Every time you check a p-value mid-experiment and reserve the right to stop early, you inflate your Type I error rate well beyond the nominal α = 0.05. Alpha spending functions are the principled solution.

The Peeking Problem

...
🔒

Continue reading

Sign in to access the full article.