CRO Glossary
Bonferroni Correction
A statistical adjustment that lowers the significance threshold when testing multiple hypotheses, to control the risk of false positives.
When you run several tests or check many metrics in a single experiment, the chance that at least one shows a 'significant' result purely by luck goes up sharply. The Bonferroni correction addresses this by dividing your target significance level (commonly 0.05) by the number of comparisons you're making, producing a stricter threshold that each individual test must clear.
Practitioners reach for this when an experiment has many secondary metrics, several variants, or multiple segments being checked at once. Without a correction, a test with 20 metrics has a good chance of showing at least one false winner even if nothing real is happening — the correction pushes teams to demand stronger evidence before declaring a win.
For example, if a test tracks 10 metrics and you want an overall 5% false-positive rate, Bonferroni would require each metric to hit p < 0.005 rather than p < 0.05 before you trust it. The trade-off is reduced statistical power: legitimate effects become harder to detect, which is why some teams prefer less conservative alternatives like controlling the false discovery rate instead.
It's a blunt but widely understood tool, and it pairs naturally with the multiple comparisons problem — the correction is essentially the standard fix for that underlying issue.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply bonferroni correction.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.