CRO Glossary
Survivorship Bias
The error of drawing conclusions only from users or variants that 'survived' a process, ignoring those who dropped out or failed.
Survivorship bias happens when an analysis only looks at the subset of users, sessions, or variants that made it through some filter — and ignores the ones that didn't — leading to misleadingly rosy conclusions. In experimentation, this often shows up when teams analyze only users who completed checkout, or only look at retained customers, without accounting for everyone who churned or bounced along the way.
A classic CRO example: a team studies behavior of users who reached the final confirmation page to understand 'what makes a good customer journey,' but that population has already self-selected for success — the friction points that caused most visitors to drop off earlier are invisible in that dataset. Any optimization based on it risks polishing an already-working path while ignoring the leaks upstream.
It also appears in post-hoc experiment analysis: if an underperforming test is stopped early and excluded from a broader report on 'what tests tend to win,' the remaining data overstates the historical win rate and skews future prioritization. Guarding against survivorship bias means deliberately including failed experiments, dropped-off users, and abandoned carts in the denominator, not just the ones who stuck around.
It's related to but distinct from attrition bias — survivorship bias is about how conclusions get warped by looking only at the surviving sample, rather than about the mechanism causing differential dropout itself.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply survivorship bias.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.