Winner's Curse

The winner's curse happens because, among many variants or many peeks at data, the one that happens to look best at the moment you stop testing is partly winning due to random noise in its favor, not just true underlying improvement. When you then implement that 'winning' variant permanently, the real-world lift is often smaller than what the experiment reported, sometimes disappointingly so.

This is closely related to regression to the mean and is exacerbated by the peeking problem: the more variants you test simultaneously, or the more times you check results before a test is fully powered, the more likely you are to catch a variant during a lucky streak. It's also connected to the false discovery rate issue in multi-test programs, where running many experiments increases the odds that some 'wins' are false positives.

A practical defense is validating big wins with a follow-up confirmation test before rolling out at scale, and being appropriately skeptical of extremely large effect sizes from a single test, especially in multi-armed comparisons or multivariate testing with many combinations. Reporting a confidence interval alongside the point estimate, rather than just the headline lift number, also helps teams calibrate expectations rather than overselling the result.