All terms

CRO Glossary

Experiment Decay

The gradual erosion of a winning test's uplift over time after it is fully rolled out.

Experiment decay describes the phenomenon where a variant that showed a strong lift during testing delivers a smaller benefit, or none at all, once it's permanently implemented and time passes. The initial measured uplift was real at the time, but the magnitude fades as market conditions, user populations, or competitive context shift.

This is distinct from novelty effect, which fades quickly within days or weeks as users get used to a new element. Experiment decay operates on a longer horizon, sometimes months, and can stem from seasonal shifts, changes in traffic mix, competitors copying the change, or the original effect simply regressing once it's no longer being compared head-to-head against a live control.

A concrete example: a pricing page redesign lifts conversion 12% during a four-week test. Six months after full rollout, year-over-year analysis shows only a 3% sustained improvement, with the gap explained by seasonal traffic differences and a competitor's similar redesign launched shortly after.

Teams guard against experiment decay by periodically re-validating big wins with fresh holdout groups, tracking long-term metric trends after rollout rather than assuming the test result is permanent, and being cautious about stacking multiple 'winning' changes without re-testing the combined effect.

Related terms

See this in the wild

ABWatcher watches how top teams apply experiment decay.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.