CRO Glossary
Carryover Effect
Bias in a test's results caused by residual influence from a prior experiment or treatment that hasn't fully worn off.
A carryover effect happens when users who were exposed to a previous experiment still behave differently because of that exposure, even after the earlier test has ended and a new one has begun. Instead of measuring the new treatment in isolation, you end up measuring some blend of the old and new effects, which can inflate, deflate, or completely mask the true impact of what you're currently testing.
This is a common risk when teams reuse the same audience pool across back-to-back experiments without a cooldown period, or when cookies and local storage retain a user's prior variant assignment. It's especially problematic for tests involving pricing, onboarding flows, or anything that changes a user's mental model of the product, since those effects can linger for days or weeks after exposure.
For example, if you ran a test last month that trained users to expect a discount banner, and this month you test a new checkout flow on the same returning visitors, some of the checkout lift or drop you observe may actually be residual behavior from the discount test, not the checkout change itself.
Practitioners guard against carryover by resetting or excluding recently-exposed users, spacing out sequential experiments on overlapping populations, and checking new-visitor-only segments as a sanity check against returning-visitor segments.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply carryover effect.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.