CRO Glossary
Power Analysis
A pre-test calculation that determines the sample size needed to reliably detect a given effect size.
Power analysis answers a deceptively simple question before you launch anything: how much traffic and time do you need to trust the result? It combines your baseline conversion rate, the smallest effect you actually care about detecting, your chosen significance threshold, and your desired statistical power (commonly 80%) into a required sample size per variant. Skip this step and you risk two failure modes — stopping too early on noise, or running a test so underpowered it could never have found a real effect even if one existed.
Practitioners care about this because traffic is finite and every test competes for the same pool of visitors. Running a power analysis upfront forces an honest conversation: if the site only gets 2,000 monthly checkouts, a test looking for a 2% relative lift on a metric with high variance may need to run for six months — probably not worth it. Better to either target a bigger, more detectable effect or find a higher-traffic page to test on instead.
A concrete example: a SaaS signup page converts at 5%. The team wants to detect a 10% relative lift (5% to 5.5%) with 80% power and 95% confidence. A power calculation shows they need roughly 30,000 visitors per variant — at current traffic, that's eight weeks. Knowing this before launch lets them decide whether to widen the MDE, pool with a similar page, or accept the longer runtime, rather than discovering the problem three weeks in when the confidence interval is still uselessly wide.
Power analysis is closely tied to MDE and sample size calculators, but it's the broader planning discipline — the calculation itself is just one input into a go/no-go decision about whether a test is worth running at all.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply power analysis.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.