All terms

CRO Glossary

Attrition Bias

A distortion in test results caused by users dropping out of one variant's experience at a different rate than the other's.

Attrition bias occurs when participants leave an experiment — by abandoning the flow, closing the tab, or simply not completing the funnel step being measured — at systematically different rates between the control and the variant. If the people who drop out differ in some meaningful way from those who stay, the remaining sample is no longer comparable across groups, and the final comparison of conversion rates is skewed even though randomization was done correctly at the start.

This matters because standard significance tests assume the groups being compared are still equivalent except for the treatment itself. If a confusing new checkout variant scares off price-sensitive users early, the users who persist all the way to purchase might just be less price-sensitive on average, making the variant look artificially strong at the final conversion step while the top-of-funnel damage goes unnoticed.

A classic example: a multi-step signup redesign that adds an extra screen. Some users bail out at that new screen who would have completed the old, shorter flow. If the analysis only looks at conversion rate among users who reached the final step, it ignores everyone who was lost at the new screen, inflating the apparent win. The fix is to measure conversion from a fixed, consistent entry point for both arms — everyone who entered the funnel, not everyone who survived to a later step — and to watch drop-off rate at each step as its own guardrail metric.

Related terms

See this in the wild

ABWatcher watches how top teams apply attrition bias.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.