CRO Glossary
Segment Analysis
Breaking down experiment results by user subgroups (like device, geography, or new vs. returning) to see if effects vary.
Segment analysis takes an experiment's overall result and slices it by dimensions such as device type, traffic source, geography, browser, or new versus returning visitor status, to check whether the treatment effect is consistent across groups or concentrated in just one. It's a useful diagnostic step after a test concludes, helping teams understand not just whether something worked, but for whom and why.
The major risk with segment analysis is that testing many segments after the fact dramatically increases the odds of finding a spurious 'significant' result purely by chance — closely related to the multiple-comparisons issue behind false discovery rate. A result that only shows up in, say, Safari users on tablets in one region should be treated as a hypothesis to test again, not a proven finding, unless it was specified as a segment of interest before the test ran.
A well-known real-world pitfall connects here too: aggregate results can sometimes hide a reversal that only appears when segmented by a third variable, which is the basis of Simpson's paradox. For example, a redesign might show a flat overall conversion result, but segment analysis could reveal it significantly helped mobile users while significantly hurting desktop users — information that gets lost if you only look at the top-line number.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply segment analysis.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.