CRO Glossary
Type II Error
Failing to detect a real effect in a test, incorrectly concluding there is no difference between variants when one actually exists.
A Type II error happens when an experiment concludes there's no meaningful difference between control and variant, but a real difference actually exists in the underlying population. It's the mirror image of a Type I error (a false positive): instead of seeing an effect that isn't there, you miss an effect that is there.
The most common cause is underpowered tests — too little traffic, too short a duration, or too small an effect size relative to the sample collected. Teams chasing quick answers often end tests early or run them on low-traffic pages, which inflates the risk of a Type II error. The probability of avoiding this mistake is captured by statistical power (commonly targeted at 80%), meaning even well-designed tests accept some chance of missing real winners.
For example, imagine a new checkout button copy genuinely improves conversion by 2%, but the test only runs for a few days on a low-traffic page. The sample size may be too small to reach significance, so the team concludes 'no significant difference' and discards a change that would have helped. Because Type II errors are silent — no alert fires, nothing looks obviously wrong — they quietly shrink a team's long-run experimentation win rate. Running a sample size calculator before launch and resisting the urge to stop tests early are the main defenses.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply type ii error.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.