Statistical Significance in A/B Testing: Why Sample Size Matters
Most A/B "winners" are underpowered noise. Here's the math on sample size, MDE, and why calling significance early quietly destroys revenue.
Sam Lee
Data Analyst · · Updated
Somewhere on your team's experiment log is a test that shipped with an n of 800 per arm, a p-value of 0.04, and a victory lap in Slack. Statistically, that result had maybe a coin-flip's chance of replicating. This is the single most common failure mode in CRO: teams run real, well-designed tests, then stop them the moment the dashboard turns green, without ever asking whether the sample size could actually detect the effect they're claiming.
We watch live experiments across 1,000+ high-converting sites at ABWatcher, and the pattern is consistent — hero copy, banner, and social-proof tests running in the wild right now, from Magic Spoon to Buffer to AB Tasty itself. The tests are legitimate. The question this article answers is how you'd know, statistically, whether their results (or yours) are trustworthy.
TL;DR
- A 5% relative lift on a 10% baseline conversion rate requires ~31,000 visitors per arm at 80% power and 95% confidence.
- Stopping a test early at "p < 0.05" inflates false-positive rates from 5% up to 20-30% with repeated peeking (Optimizely-style continuous monitoring problem).
- Magic Spoon was running a hero A/B test detected 29 May 2026, swapping full product imagery for a nutrition-callout card layout.
- Buffer split hero subheadline copy across Chrome and Firefox simultaneously, detected 26 Aug 2026 — a clean cross-browser test, not rendering noise.
- AB Tasty tested its own hero headline ending — "personalization" vs. "conversions" — detected 26 Aug 2026.
- Practical significance (is the lift big enough to matter) and statistical significance (is the lift real) are answering two different questions — you need both.
What sample size do you actually need to detect a real effect?
You need a sample size calculated from your baseline conversion rate, your minimum detectable effect (MDE), and your desired statistical power — not "whatever traffic shows up in two weeks."
The standard formula for a two-proportion test, per Cochran's sampling techniques, boils down to:
n = 16 × p(1-p) / d²
where p is your baseline conversion rate and d is the absolute minimum difference you want to detect, at roughly 80% power and 95% confidence. Run the numbers:
| Baseline CVR | MDE (relative) | Absolute lift (d) | Required n per arm |
|---|---|---|---|
| 10% | 20% | 2.0 pp | ~1,900 |
| 10% | 10% | 1.0 pp | ~7,500 |
| 10% | 5% | 0.5 pp | ~31,000 |
| 3% | 10% | 0.3 pp | ~30,300 |
| 3% | 5% | 0.15 pp | ~121,000 |
Notice what happens as your baseline conversion rate drops and your MDE shrinks: required sample size explodes. A fintech signup flow converting at 3%, chasing a realistic 5% relative lift, needs 121,000 visitors per arm — 242,000 total. Most mid-market SaaS sites don't have that much monthly traffic to a single page, which is exactly why so many "significant" results on low-traffic pages are actually noise dressed up in a green banner.
SplitMetrics' guidance on mobile A/B testing makes the same point from the App Store angle: with lower daily impression volumes, teams routinely under-power creative tests and then over-interpret a one-week swing.
Why does statistical significance differ from practical significance?
Statistical significance tells you an effect is probably not zero; practical significance tells you whether that effect is big enough to justify shipping it — and they are answered by different numbers entirely.
Say you run a test for six weeks and hit p = 0.03 on a lift from 10.0% to 10.3% conversion. That's statistically significant if your sample was large enough (per the table above, you'd need roughly 140,000 visitors per arm to reliably detect a 0.3-point absolute move on a 10% base). But is a 3% relative lift worth a permanent code change, a design system deviation, and ongoing maintenance? On a $50M ARR pricing page, a 3% lift is worth roughly $1.5M annualized — clearly worth shipping. On a $2M ARR page, that same 3% is $60K — probably not worth the engineering backlog slot ahead of three other roadmap items.
This is where AB Tasty's own hero test is instructive. Detected 26 Aug 2026, the AB Tasty homepage headline split "AI-powered personalization" against "AI-powered conversions" as the final word. A single-word swap like this typically produces small effect sizes — plausibly in the 1-3% relative range on click-through to demo request. To detect that reliably, AB Tasty would need tens of thousands of sessions per arm on that specific headline exposure, not just aggregate site traffic. Single-word copy tests are exactly the category most likely to be underpowered if run for only a week or two.
How do teams accidentally inflate false positives without realizing it?
Peeking at results daily and stopping as soon as you cross p < 0.05 turns a nominal 5% false-positive rate into an actual rate of 20-30% or higher.
This is the "optional stopping" problem, and it's the single biggest reason "winning" variants fail to replicate when re-tested. Every time you check a dashboard and make a stop/continue decision based on the p-value at that moment, you're running an implicit multiple-comparisons test against your own significance threshold. Check daily over four weeks, and you've effectively run ~28 significance tests, not one — and standard 95% CI math assumes exactly one.
The fix is either (a) fix your sample size and calendar duration in advance using the formula above and refuse to call the test until you hit it, or (b) use a sequential testing method (like an always-valid p-value or Bayesian stopping rule) explicitly designed for repeated looks. Optibase's breakdown of sample size mechanics frames this as the three interlocking inputs — baseline rate, MDE, and confidence level — that must be locked before launch, not adjusted mid-flight to make a result look better.
Consider Atlas Coffee Club's footer test, detected 18 Aug 2026 on Atlas Coffee Club: some visits show an email-capture CTA ("Your first bag free") in the slot below checkout, others show a five-star testimonial from a named customer. Social proof vs. direct-offer tests like this often show early "winners" driven by a handful of high-intent sessions in the first 48 hours — exactly the window where sample size is smallest and variance is highest. Without a pre-registered sample size target, an early lead for either variant is close to a coin flip.
How do multi-browser and multi-variant detections help you sanity-check a live test?
Seeing the same split reproduce across independent browser engines is a useful, low-cost confirmation that you're looking at a genuine assignment split rather than caching or rendering artifacts — but it says nothing about whether the sample is big enough to trust the result.
Buffer's hero subheadline test, detected 26 Aug 2026 on Buffer, is a clean example: "Connected to every platform and tool you use" appeared in visits 1, 3, 4, and 5, while "Works with every platform you post to, and plugs into your favorite tools" appeared in visits 6 and 7 — and the split held across both Chrome and Firefox on each side. That cross-browser consistency rules out the "it's just a rendering bug" explanation. Andi's announcement banner test, detected 15 Aug 2026 on Andi, shows the same pattern: the green banner appeared in Chrome, Edge, and one Firefox visit, and was absent in Safari and another Firefox visit — a real assignment split, not a browser quirk.
But confirming the split is real is a different question from confirming the result is powered. A homepage subheadline test with strong semantic difference (functional "connected to every platform" vs. benefit-driven "plugs into your favorite tools") could easily produce a 2-4% relative lift on secondary engagement metrics — which per the table above needs somewhere between 8,000 and 30,000 sessions per arm to detect reliably, depending on the baseline metric. If Buffer's test is measuring signup clicks off a homepage getting tens of thousands of monthly visits, that's achievable within a few weeks. If it's measuring a downstream metric like paid conversion, the sample size requirement multiplies with the funnel drop-off.
What should you actually check before calling a test a winner?
Run three checks before you ship: (1) did you hit your pre-calculated sample size, (2) is the confidence interval on the lift entirely above zero and excludes a trivial effect, and (3) does the effect hold in a held-out segment.
Concretely:
- Pre-register your MDE and n. Decide the smallest lift worth shipping (say, 5% relative) before launch, plug it into the formula, and commit to a sample size — not a calendar date.
- Report the confidence interval, not just the point estimate. A "+4% lift" with a 95% CI of [-1%, +9%] is not a winner; it's a coin flip that happened to land favorably.
- Segment-check the win. A lift that only shows up in one browser, one device class, or one traffic source is a red flag for either an implementation bug or a genuinely heterogeneous effect that needs a bigger sample to confirm in aggregate.
- Watch for novelty effects. Banner and hero changes — like Andi's announcement bar or Magic Spoon's nutrition-callout hero, detected 29 May 2026 on Magic Spoon — often spike engagement in week one simply because returning visitors notice something new, then regress toward baseline by week three. Re-test at two time horizons before trusting a week-one lift.
Common questions
How long should I run an A/B test before checking significance?
Run it until you hit your pre-calculated sample size per arm, not a fixed number of days. As a floor, most teams should plan for at least one full business cycle (commonly 2 weeks) to average out day-of-week effects, and longer if your baseline conversion rate is under 5%.
What's a reasonable minimum detectable effect (MDE) to test for?
Most CRO teams target a 5-10% relative lift as their MDE, since anything smaller requires prohibitively large samples for typical traffic levels, and anything a genuine 15%+ lift is usually visible within a much smaller, faster test.
Can a test be statistically significant but not worth shipping?
Yes — a statistically significant 0.5% relative lift on a low-revenue page may not clear the bar for engineering effort or design-system deviation, even at 99% confidence. Always convert the lift into an annualized dollar figure against your traffic and AOV before prioritizing the ship.
Why do so many "winning" variants fail to replicate later?
The two leading causes are optional stopping (peeking daily and stopping at the first p < 0.05) and underpowered samples relative to the true effect size, both of which inflate the real false-positive rate well above the nominal 5%.
Does cross-browser consistency prove a test is trustworthy?
It proves the split is a genuine experiment assignment rather than a rendering artifact, which rules out one category of false detection — but it says nothing about statistical power. You still need to check sample size and confidence intervals separately.
If there's one thing to add to your team's experiment checklist this sprint, it's this: before you write "MDE" into the ticket, calculate the required sample size using your actual baseline conversion rate, and put that number — not a launch date — as the stopping condition. It's a 10-minute spreadsheet exercise that will quietly kill more false "wins" than any amount of test-design sophistication.
Read next
More from the ABWatcher blog
Nov 22, 2026
How to Calculate Confidence Intervals for Conversion Rate Lift
A practical guide to computing confidence intervals for A/B test conversion lift — frequentist and Bayesian methods, worked examples, and how to report ranges instead of fake certainty.
Nov 21, 2026
Why Your Conversion Rate Varies by Traffic Source (And What to Do About It)
A diagnostic guide to conversion rate variance across organic, paid, and referral traffic — how to separate real signal from source-quality noise before you spin up a new test.
Nov 20, 2026
Why Your A/B Test Winners Don't Replicate Across Traffic Sources
A test winner that holds for blended traffic often fails for a specific channel. Here's why, with real examples, and how to validate before you scale.
Nov 19, 2026
Session Recording for CRO: What to Watch When Users Bounce
A practical, intro-level guide to reading session replays for hidden friction and high-intent signals — plus how to turn what you see into testable hypotheses.
Nov 18, 2026
How to Build a Winning Landing Page Without A/B Testing
No traffic for statistical significance? Here's a design-led framework for building high-converting landing pages using proven patterns, not guesswork.
Nov 17, 2026
How to Find Your True Conversion Bottleneck (Not Just the Waterfall)
A step-by-step method for pinpointing the real cause of funnel drop-off using session recordings, heatmaps, and field-level friction data — before you waste a test cycle.
From the glossary
Vocabulary the article uses. See all terms →
- Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
- Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
- Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
- Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
- Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
- Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See more like this
ABWatcher catches A/B tests like this across 1,000+ brands.
Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.