CRO Glossary
Sample Size Calculator
A tool that estimates how many visitors or conversions an experiment needs to detect a given effect reliably.
A sample size calculator takes inputs like your baseline conversion rate, the minimum detectable effect you care about, your desired statistical power, and significance threshold, and outputs how many users (or sessions) you need per variant before you can trust the result. It exists because intuition is a poor guide here: small effects on low-traffic pages can require tens of thousands of visitors per arm, while large effects on high-traffic pages might only need a few hundred.
Teams use these calculators before launching a test, not after, to avoid the trap of running underpowered experiments that produce noisy, inconclusive reads. Skipping this step is one of the most common reasons CRO programs report high 'win rates' that don't replicate — the tests were never powered to detect what they claimed to find.
For example, if your checkout page converts at 3% and you want to detect a 10% relative lift (to 3.3%) with 80% power, a calculator might tell you that you need roughly 30,000 visitors per variant. If your site only gets 5,000 visitors a month, you now know upfront that the test needs to run for two months minimum, or that you should test a bolder change instead.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply sample size calculator.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.