CRO Glossary
Winner's Curse
The tendency for the observed effect size of a 'winning' variant to be inflated compared to its true long-term effect.
The winner's curse happens because, among many variants or many peeks at data, the one that happens to look best at the moment you stop testing is partly winning due to random noise in its favor, not just true underlying improvement. When you then implement that 'winning' variant permanently, the real-world lift is often smaller than what the experiment reported, sometimes disappointingly so.
This is closely related to regression to the mean and is exacerbated by the peeking problem: the more variants you test simultaneously, or the more times you check results before a test is fully powered, the more likely you are to catch a variant during a lucky streak. It's also connected to the false discovery rate issue in multi-test programs, where running many experiments increases the odds that some 'wins' are false positives.
A practical defense is validating big wins with a follow-up confirmation test before rolling out at scale, and being appropriately skeptical of extremely large effect sizes from a single test, especially in multi-armed comparisons or multivariate testing with many combinations. Reporting a confidence interval alongside the point estimate, rather than just the headline lift number, also helps teams calibrate expectations rather than overselling the result.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply winner's curse.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.