CRO Glossary
Attrition Bias
A distortion in test results caused by users dropping out of one variant's experience at a different rate than the other's.
Attrition bias occurs when participants leave an experiment — by abandoning the flow, closing the tab, or simply not completing the funnel step being measured — at systematically different rates between the control and the variant. If the people who drop out differ in some meaningful way from those who stay, the remaining sample is no longer comparable across groups, and the final comparison of conversion rates is skewed even though randomization was done correctly at the start.
This matters because standard significance tests assume the groups being compared are still equivalent except for the treatment itself. If a confusing new checkout variant scares off price-sensitive users early, the users who persist all the way to purchase might just be less price-sensitive on average, making the variant look artificially strong at the final conversion step while the top-of-funnel damage goes unnoticed.
A classic example: a multi-step signup redesign that adds an extra screen. Some users bail out at that new screen who would have completed the old, shorter flow. If the analysis only looks at conversion rate among users who reached the final step, it ignores everyone who was lost at the new screen, inflating the apparent win. The fix is to measure conversion from a fixed, consistent entry point for both arms — everyone who entered the funnel, not everyone who survived to a later step — and to watch drop-off rate at each step as its own guardrail metric.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply attrition bias.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.