Multiple Comparisons Problem

The multiple comparisons problem describes how statistical significance thresholds behave differently when you run many tests simultaneously instead of one. If you check 20 metrics at a 5% significance threshold, you'd expect roughly one of them to look 'significant' purely by chance, even if nothing real is happening. The more comparisons you make — more metrics, more segments, more variants — the higher the chance that at least one shows a false positive.

This is a constant hazard in CRO because dashboards often report dozens of secondary metrics alongside the primary one, and it's tempting to declare a win on whichever metric moved. Without correction, a test with no real effect on conversion can still produce a 'significant' lift in some slice — mobile users in Ohio, say — that's just noise.

A concrete example: a team runs a pricing page test and looks at conversion rate, average order value, bounce rate, time on page, and six device/browser segments — sixteen comparisons total. One segment shows a 'significant' 15% lift. Reported in isolation, it looks like a discovery; in context, it's roughly what you'd expect from chance alone given how many cuts were checked.

Practitioners guard against this by designating one primary metric before the test starts, treating secondary metrics and segment cuts as exploratory/hypothesis-generating rather than confirmatory, and applying corrections (like Bonferroni or false discovery rate control) when multiple formal comparisons are unavoidable.