CRO Glossary
Multiple Comparisons Problem
The increased risk of false positives that arises when many metrics, segments, or variants are tested at once.
The multiple comparisons problem describes how statistical significance thresholds behave differently when you run many tests simultaneously instead of one. If you check 20 metrics at a 5% significance threshold, you'd expect roughly one of them to look 'significant' purely by chance, even if nothing real is happening. The more comparisons you make — more metrics, more segments, more variants — the higher the chance that at least one shows a false positive.
This is a constant hazard in CRO because dashboards often report dozens of secondary metrics alongside the primary one, and it's tempting to declare a win on whichever metric moved. Without correction, a test with no real effect on conversion can still produce a 'significant' lift in some slice — mobile users in Ohio, say — that's just noise.
A concrete example: a team runs a pricing page test and looks at conversion rate, average order value, bounce rate, time on page, and six device/browser segments — sixteen comparisons total. One segment shows a 'significant' 15% lift. Reported in isolation, it looks like a discovery; in context, it's roughly what you'd expect from chance alone given how many cuts were checked.
Practitioners guard against this by designating one primary metric before the test starts, treating secondary metrics and segment cuts as exploratory/hypothesis-generating rather than confirmatory, and applying corrections (like Bonferroni or false discovery rate control) when multiple formal comparisons are unavoidable.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply multiple comparisons problem.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.