How to Calculate Statistical Power for A/B Tests: A Practical Guide
A founder's walkthrough of statistical power in A/B testing — what it actually means, how to calculate it, and why underpowered tests quietly kill good ideas.
Jordan Reeves
Founder & Operator · · Updated
I've killed features that were actually winning. Not because the idea was bad, but because the test was underpowered and I read a flat result as "no effect" when it was really "we never had enough visitors to see the effect." That mistake cost one team I worked with about four months of a roadmap quarter, because we shelved a checkout change that a competitor later shipped successfully. Statistical power is the concept that would have saved us, and almost nobody explains it in a way a product manager can use on a Tuesday afternoon.
TL;DR
- Standard statistical power target is 80%, meaning a 20% chance of missing a real winning variant (a false negative).
- Power depends on four variables — baseline conversion rate, minimum detectable effect (MDE), significance level (α, usually 5%), and sample size — fix three and the fourth is determined, per DRIP's power analysis.
- A 3% baseline conversion rate test detecting a 15% relative lift needs roughly 23,000 visitors per variation (~46,000 total), according to this sample size walkthrough.
- High-impact tests (pricing, checkout, onboarding) often justify bumping power to 90-95% despite the larger sample cost.
- ABWatcher detected live layout, hero, and routing experiments at Zillow, AB Tasty, and Beyond Meat in 2026 — all tests that only mean something if the underlying sample was powered correctly.
What is statistical power, actually?
Statistical power is the probability your test detects a real effect when one truly exists. Flip it around: an 80%-powered test still has a 20% chance of missing a genuine winner and reporting "no significant difference" — a false negative, also called a Type II error. This is distinct from your significance level (α), which governs false positives — the chance you declare a winner when there isn't one. Most teams obsess over α and never check power, which is like proofreading for typos while leaving the plot holes in.
Here's the part that's easy to miss: a "flat" test result doesn't mean the variant doesn't work. It might mean your test was never capable of showing you that it does.
Why does low power quietly kill good ideas?
Low power produces false negatives that look identical to true negatives, so teams discard winning variants without knowing it. If you ran 4,000 visitors per arm when you needed 23,000, your test wasn't wrong — it was blind. You genuinely did not collect enough signal to separate a real 15% lift from noise. The DigitalApplied CRO guide puts the standard baseline at 80% power, meaning even a well-run test misses one in five real wins. For anything touching revenue — pricing pages, checkout flows, trial-to-paid prompts — that 20% miss rate is expensive enough to justify pushing power to 90% or 95%, at the cost of a bigger sample.
This is exactly the gap between "how long should I run this test" and "how confident can I be in a null result." Duration tells you when to stop. Power tells you whether stopping with a flat result actually means anything.
How do you calculate the sample size you need?
Sample size is driven by four interlocking inputs: baseline conversion rate, minimum detectable effect, significance level, and target power. The formula, per Redclawey's 2026 experimentation guide, is:
n = (Zα/2 + Zβ)² × 2 × p × (1 − p) / (p1 − p2)²
You don't need to compute Z-scores by hand — tools like Evan Miller's calculator or Optimizely's sample size calculator do it instantly. But understanding the inputs matters more than the arithmetic:
| Input | What it means | Typical default |
|---|---|---|
| Baseline conversion rate (p) | Your current rate before any change | Measured from last 30-90 days of data |
| Minimum detectable effect (MDE) | Smallest lift worth caring about | 10-20% relative lift |
| Significance level (α) | Tolerance for false positives | 5% |
| Statistical power (1 − β) | Tolerance for false negatives | 80% (90-95% for high-stakes tests) |
A worked example from the Medium sample size breakdown: a product page converting at 3% add-to-cart, targeting a 15% relative lift (3.0% → 3.45%) at 80% power, needs roughly 23,000 visitors per variation — about 46,000 total. Halve your baseline rate or your MDE, and your required sample roughly quadruples. That's the part that surprises first-time experimenters: small effects on low-traffic pages are brutally expensive to detect reliably.
What does this look like in live tests?
Real companies are running exactly this trade-off right now, whether or not they publish their power calculations. ABWatcher caught Zillow running an Optimizely experiment on its BuyAbility flow, detected 24 May 2026 — the 5-step green process box (Calculate BuyAbility, Get pre-approved, Make an offer, Apply for a mortgage, Close on your home) appeared in visits 1 and 3 but was absent in visit 2, a classic on/off section toggle. A test like that, sitting on a core conversion path, is exactly the kind of change where you'd want 90% power — a missed read on a mortgage-funnel element isn't a rounding error, it's lost revenue at scale.
We also saw AB Tasty running a server-side split test detected 17 September 2026, where visitors landed on two entirely different destinations — wingify.com with soft pastel branding and "Every visitor is different" messaging, versus abtasty.com's neon lime/purple "Push the limits of experience optimization with AI-powered conversions" hero. That's a dramatic MDE by design: when your variants are this different, you don't need the eye-watering sample sizes a 2% tweak would require, because a bigger true effect is easier to detect with fewer visitors — the inverse relationship between MDE and required sample size in action.
And Beyond Meat ran a hero and carousel experiment detected 13 June 2026, where one visit (visit 5, Chrome) showed "Plants have superpowers." with a plant close-up and a different "Plants are nutrition powerhouses." carousel slide, against a control showing an email modal overlay and "We love clean protein." messaging. A full hero and messaging swap like this is a high-MDE test too — good, because top-of-funnel ecommerce traffic splits can still take weeks to reach a clean sample on a lower-traffic variant.
How do you decide between running longer or accepting more risk?
You weigh the cost of a longer test against the cost of a false negative on that specific page. If you're testing a homepage hero with huge traffic, hitting 23,000+ per arm might take days — run it properly. If you're testing a checkout micro-copy change on a lower-traffic enterprise funnel, the same sample size might take three months, and that's when teams start cutting corners: shrinking the MDE requirement, lowering power to 70%, or peeking early and stopping at the first significant blip. All three decisions quietly increase your false negative or false positive rate. The honest move is to decide upfront, before you see any data, what power and MDE you're willing to accept — and if the traffic can't support it in a reasonable window, that's a signal to test something with a bigger expected effect instead, the way AB Tasty's two-destination split or Beyond Meat's full hero swap did.
Common questions
What's a good statistical power target for A/B tests?
80% is the accepted industry standard, meaning a 20% chance of missing a real effect. For high-impact surfaces like pricing, checkout, or onboarding flows, many teams push to 90-95% power, accepting a larger required sample size in exchange for a lower risk of discarding a genuine winner.
How is statistical power different from statistical significance?
Significance (α) controls your false positive rate — declaring a winner that isn't real, typically capped at 5%. Power controls your false negative rate — missing a winner that is real, typically capped at 20% (80% power). They govern opposite types of error and both need to be set deliberately.
Can I increase power without collecting more traffic?
Only by accepting a larger minimum detectable effect — meaning you'll only reliably catch bigger wins, not subtle ones. Otherwise, power, sample size, MDE, and significance level are mathematically linked; improving one without more traffic means loosening another, usually your MDE or α threshold.
Why did my A/B test show no significant difference?
Either the variant genuinely performs the same as control, or your test lacked the power to detect a real but smaller-than-expected effect. Before concluding "no difference," check whether your actual sample size matched what a proper power calculation required for your baseline rate and target MDE.
How long should I run a test to hit 80% power?
It depends entirely on your daily traffic relative to the required sample size — there's no fixed day count. Calculate required sample size first using your baseline conversion rate and target MDE, then divide by your expected daily traffic per variant to get a realistic duration, as outlined in Redclawey's experimentation guide.
What I would actually do next: before your next experiment kicks off, run the sample size math first using your real baseline rate and the smallest lift you'd actually act on — not after the test stalls at week two and someone asks "is this done yet?" If the required sample doesn't fit your traffic in a timeframe you can live with, either redesign the test to target a bigger effect (like a full hero swap, not a button color) or accept you're running an underpowered exploratory test and label the result accordingly instead of calling it a loss.
Read next
More from the ABWatcher blog
Nov 22, 2026
How to Calculate Confidence Intervals for Conversion Rate Lift
A practical guide to computing confidence intervals for A/B test conversion lift — frequentist and Bayesian methods, worked examples, and how to report ranges instead of fake certainty.
Nov 21, 2026
Why Your Conversion Rate Varies by Traffic Source (And What to Do About It)
A diagnostic guide to conversion rate variance across organic, paid, and referral traffic — how to separate real signal from source-quality noise before you spin up a new test.
Nov 20, 2026
Why Your A/B Test Winners Don't Replicate Across Traffic Sources
A test winner that holds for blended traffic often fails for a specific channel. Here's why, with real examples, and how to validate before you scale.
Nov 19, 2026
Session Recording for CRO: What to Watch When Users Bounce
A practical, intro-level guide to reading session replays for hidden friction and high-intent signals — plus how to turn what you see into testable hypotheses.
Nov 18, 2026
How to Build a Winning Landing Page Without A/B Testing
No traffic for statistical significance? Here's a design-led framework for building high-converting landing pages using proven patterns, not guesswork.
Nov 17, 2026
How to Find Your True Conversion Bottleneck (Not Just the Waterfall)
A step-by-step method for pinpointing the real cause of funnel drop-off using session recordings, heatmaps, and field-level friction data — before you waste a test cycle.
From the glossary
Vocabulary the article uses. See all terms →
- Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
- Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
- Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
- Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
- Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
- Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See more like this
ABWatcher catches A/B tests like this across 1,000+ brands.
Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.