CRO Glossary
Experiment Decay
The gradual erosion of a winning test's uplift over time after it is fully rolled out.
Experiment decay describes the phenomenon where a variant that showed a strong lift during testing delivers a smaller benefit, or none at all, once it's permanently implemented and time passes. The initial measured uplift was real at the time, but the magnitude fades as market conditions, user populations, or competitive context shift.
This is distinct from novelty effect, which fades quickly within days or weeks as users get used to a new element. Experiment decay operates on a longer horizon, sometimes months, and can stem from seasonal shifts, changes in traffic mix, competitors copying the change, or the original effect simply regressing once it's no longer being compared head-to-head against a live control.
A concrete example: a pricing page redesign lifts conversion 12% during a four-week test. Six months after full rollout, year-over-year analysis shows only a 3% sustained improvement, with the gap explained by seasonal traffic differences and a competitor's similar redesign launched shortly after.
Teams guard against experiment decay by periodically re-validating big wins with fresh holdout groups, tracking long-term metric trends after rollout rather than assuming the test result is permanent, and being cautious about stacking multiple 'winning' changes without re-testing the combined effect.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment decay.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.