CRO Glossary
Test Duration Calculator
A tool that estimates how long an experiment must run to reach a reliable sample size given traffic and baseline conversion rate.
A test duration calculator combines your site's traffic volume, baseline conversion rate, minimum detectable effect, and desired statistical power to output an estimated number of days (or weeks) needed to run an experiment. It's essentially the sample size calculator translated into a timeline, since sample size alone doesn't tell a team when they can stop looking at results.
Practitioners rely on this before launch, not during, because deciding duration mid-test invites peeking and inflates false positive rates. A good estimate also forces a useful conversation: if the calculator says a test needs 11 weeks to detect a 5% lift, the team can decide upfront whether that's worth doing, or whether they should target a bigger change, a higher-traffic page, or a longer-tail metric instead.
For example, a checkout page getting 2,000 visitors a week with a 4% baseline conversion rate might need roughly 6 weeks to reliably detect a 10% relative lift at 80% power. If the roadmap only allows 2 weeks, the team either accepts higher risk of missing real effects or redesigns the test to target a larger, more detectable change.
A common gotcha is treating the calculator's output as a hard promise rather than a planning estimate — actual traffic can dip due to seasonality, holidays, or unrelated site changes, so most teams pad the estimate and monitor progress against it weekly rather than committing to a fixed calendar date.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply test duration calculator.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.