Type I Error

A Type I error happens when a test declares a winner that isn't really better — the apparent lift is noise, not a true effect. This is governed by your significance threshold (commonly alpha = 0.05), which is the probability of a false positive you're willing to accept on any single valid test.

Type I errors matter in CRO because they lead teams to ship changes that don't actually help, sometimes actively hurting conversion once the illusory lift disappears post-launch. They're especially likely when teams run many tests or check results repeatedly without correcting for it — each additional look or additional variant increases the cumulative chance that at least one false positive shows up somewhere.

For example, if a team runs 20 uncorrected tests at alpha = 0.05, they should expect roughly one false positive purely by chance, even if none of the changes actually work. This is why practices like pre-registration, correcting for multiple comparisons, and avoiding continuous peeking exist — they keep the real false-positive rate close to the stated threshold.

Type I error is the mirror image of Type II error (a false negative, missing a real effect). Tightening alpha to reduce Type I errors increases the risk of Type II errors, so teams have to choose a balance appropriate to the cost of being wrong in either direction.