CRO Glossary
Type II Error
Failing to detect a real effect in a test, incorrectly concluding there is no difference between variants when one actually exists.
A Type II error happens when an experiment concludes there's no meaningful difference between control and variant, but a real difference actually exists in the underlying population. It's the mirror image of a Type I error (a false positive): instead of seeing an effect that isn't there, you miss an effect that is there.
The most common cause is underpowered tests — too little traffic, too short a duration, or too small an effect size relative to the sample collected. Teams chasing quick answers often end tests early or run them on low-traffic pages, which inflates the risk of a Type II error. The probability of avoiding this mistake is captured by statistical power (commonly targeted at 80%), meaning even well-designed tests accept some chance of missing real winners.
For example, imagine a new checkout button copy genuinely improves conversion by 2%, but the test only runs for a few days on a low-traffic page. The sample size may be too small to reach significance, so the team concludes 'no significant difference' and discards a change that would have helped. Because Type II errors are silent — no alert fires, nothing looks obviously wrong — they quietly shrink a team's long-run experimentation win rate. Running a sample size calculator before launch and resisting the urge to stop tests early are the main defenses.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply type ii error.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.