CRO Glossary
Confidence Interval
A range of plausible values for a metric's true effect, given the observed data and a chosen confidence level.
A confidence interval expresses uncertainty around an estimate. If a test shows a conversion lift of 8% with a 95% confidence interval of [2%, 14%], it means that if you repeated the experiment many times, 95% of such intervals would contain the true effect — the true lift is plausibly anywhere in that range, not just the single point estimate.
Wide intervals signal you need more data or a longer test; narrow intervals signal a more precise estimate. This is closely tied to statistical power and sample size — small samples produce wide, often unhelpfully vague intervals even when a point estimate looks impressive.
A practical habit: instead of asking "is this test a winner?", ask "what does the confidence interval say?" If the interval spans both negative and positive values (e.g., -1% to +9%), the test hasn't ruled out the possibility that the change actually hurts conversion, even if the average result looks positive.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply confidence interval.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.