CRO Glossary
Type I Error
A false positive: concluding a variant beats the control when the observed difference is actually due to chance.
A Type I error happens when a test declares a winner that isn't really better — the apparent lift is noise, not a true effect. This is governed by your significance threshold (commonly alpha = 0.05), which is the probability of a false positive you're willing to accept on any single valid test.
Type I errors matter in CRO because they lead teams to ship changes that don't actually help, sometimes actively hurting conversion once the illusory lift disappears post-launch. They're especially likely when teams run many tests or check results repeatedly without correcting for it — each additional look or additional variant increases the cumulative chance that at least one false positive shows up somewhere.
For example, if a team runs 20 uncorrected tests at alpha = 0.05, they should expect roughly one false positive purely by chance, even if none of the changes actually work. This is why practices like pre-registration, correcting for multiple comparisons, and avoiding continuous peeking exist — they keep the real false-positive rate close to the stated threshold.
Type I error is the mirror image of Type II error (a false negative, missing a real effect). Tightening alpha to reduce Type I errors increases the risk of Type II errors, so teams have to choose a balance appropriate to the cost of being wrong in either direction.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply type i error.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.