CRO Glossary
Sample Size Calculator
A tool that estimates how many visitors or conversions an experiment needs to detect a given effect reliably.
A sample size calculator takes inputs like your baseline conversion rate, the minimum detectable effect you care about, your desired statistical power, and significance threshold, and outputs how many users (or sessions) you need per variant before you can trust the result. It exists because intuition is a poor guide here: small effects on low-traffic pages can require tens of thousands of visitors per arm, while large effects on high-traffic pages might only need a few hundred.
Teams use these calculators before launching a test, not after, to avoid the trap of running underpowered experiments that produce noisy, inconclusive reads. Skipping this step is one of the most common reasons CRO programs report high 'win rates' that don't replicate — the tests were never powered to detect what they claimed to find.
For example, if your checkout page converts at 3% and you want to detect a 10% relative lift (to 3.3%) with 80% power, a calculator might tell you that you need roughly 30,000 visitors per variant. If your site only gets 5,000 visitors a month, you now know upfront that the test needs to run for two months minimum, or that you should test a bolder change instead.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply sample size calculator.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.