All terms

CRO Glossary

Power Analysis

A pre-test calculation that determines the sample size needed to reliably detect a given effect size.

Power analysis answers a deceptively simple question before you launch anything: how much traffic and time do you need to trust the result? It combines your baseline conversion rate, the smallest effect you actually care about detecting, your chosen significance threshold, and your desired statistical power (commonly 80%) into a required sample size per variant. Skip this step and you risk two failure modes — stopping too early on noise, or running a test so underpowered it could never have found a real effect even if one existed.

Practitioners care about this because traffic is finite and every test competes for the same pool of visitors. Running a power analysis upfront forces an honest conversation: if the site only gets 2,000 monthly checkouts, a test looking for a 2% relative lift on a metric with high variance may need to run for six months — probably not worth it. Better to either target a bigger, more detectable effect or find a higher-traffic page to test on instead.

A concrete example: a SaaS signup page converts at 5%. The team wants to detect a 10% relative lift (5% to 5.5%) with 80% power and 95% confidence. A power calculation shows they need roughly 30,000 visitors per variant — at current traffic, that's eight weeks. Knowing this before launch lets them decide whether to widen the MDE, pool with a similar page, or accept the longer runtime, rather than discovering the problem three weeks in when the confidence interval is still uselessly wide.

Power analysis is closely tied to MDE and sample size calculators, but it's the broader planning discipline — the calculation itself is just one input into a go/no-go decision about whether a test is worth running at all.

Related terms

See this in the wild

ABWatcher watches how top teams apply power analysis.

Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.