CRO Glossary
Holdout Group
A segment of users deliberately kept out of a new feature or campaign, used as a long-term baseline for measuring impact.
A holdout group is a slice of traffic or users that continues to experience the old version of a product, or receives no treatment at all, even after a feature has been rolled out to everyone else. Unlike a standard A/B test control group, which usually only exists for the duration of the experiment, a holdout is often kept in place for weeks or months after launch to measure the feature's true incremental impact over a longer horizon.
Holdouts are especially valuable for catching effects that a short test window would miss, such as gradual behavior changes, long-term retention effects, or the decay of a novelty effect. They're also used at the marketing-channel level — for instance, withholding a small percentage of users from receiving any retargeting ads to measure the channel's true incremental contribution versus what would have happened anyway.
A concrete example: after rolling out a new recommendation algorithm to 95% of users, a team might keep 5% on the old algorithm for three months to track whether the initial lift in engagement holds up, grows, or fades over time.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply holdout group.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.