CRO Glossary
Segment Analysis
Breaking down experiment results by user subgroups (like device, geography, or new vs. returning) to see if effects vary.
Segment analysis takes an experiment's overall result and slices it by dimensions such as device type, traffic source, geography, browser, or new versus returning visitor status, to check whether the treatment effect is consistent across groups or concentrated in just one. It's a useful diagnostic step after a test concludes, helping teams understand not just whether something worked, but for whom and why.
The major risk with segment analysis is that testing many segments after the fact dramatically increases the odds of finding a spurious 'significant' result purely by chance — closely related to the multiple-comparisons issue behind false discovery rate. A result that only shows up in, say, Safari users on tablets in one region should be treated as a hypothesis to test again, not a proven finding, unless it was specified as a segment of interest before the test ran.
A well-known real-world pitfall connects here too: aggregate results can sometimes hide a reversal that only appears when segmented by a third variable, which is the basis of Simpson's paradox. For example, a redesign might show a flat overall conversion result, but segment analysis could reveal it significantly helped mobile users while significantly hurting desktop users — information that gets lost if you only look at the top-line number.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply segment analysis.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.