CRO Glossary
Evan Miller Sample Size
A common shorthand for using Evan Miller's online calculator to estimate the sample size needed for a valid A/B test.
Practitioners often refer casually to "running it through Evan Miller" as a stand-in for doing a proper pre-test power calculation. The tool takes a baseline conversion rate, a minimum detectable effect, statistical power, and significance threshold, and returns the number of visitors (or conversions) needed per variant before the test can reliably detect the effect you're looking for.
The reason this matters is that teams frequently launch tests without checking whether their traffic can actually support the effect size they hope to see. A test that needs 400,000 visitors per arm to detect a 2% relative lift will simply never reach significance on a page that gets 5,000 visitors a month — no matter how long it runs.
For example, a team redesigning a checkout page with a 3% baseline conversion rate might plug in an MDE of 10% relative lift, 80% power, and a 95% confidence level, and learn they need roughly 40,000 users per variant. If their site only gets 10,000 checkout visits a month, they now know upfront to either wait longer, target a bigger effect, or test on a higher-traffic page instead.
This kind of calculation is essentially the same math behind formal sample size calculators, but the name has become a genericized reference point in the industry because Miller's calculator was one of the first widely used, free, and simple tools for this purpose.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply evan miller sample size.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.