Stop Calling the First Significant Day a Win

Continuously monitoring an A/B test and stopping as soon as the p-value drops below 0.05 dramatically inflates the false-positive rate. A simulation demonstrates this common practice can raise the rate from an intended 5% to nearly 28%, meaning many declared wins are simply statistical noise. This issue of "peeking" also inflates the measured effect size of true winners, leading to exaggerated performance estimates. The analysis compares valid alternatives that control for this error, including fixed-sample designs, group-sequential methods, and always-valid p-values.