The A/B Testing Sample Size Calculator: How Many Visitors You Actually Need

Most failed A/B tests don't fail because the hypothesis was wrong. They fail because the test never had enough visitors to detect the effect it was looking for. A team runs a homepage redesign for three weeks, sees a "directional" 6% lift that never reaches significance, and ships it anyway — or worse, kills a genuinely good idea because the data looked flat. In both cases, the root cause is the same: nobody ran the sample size math before hitting launch.

This is the least glamorous step in conversion optimization and the one most often skipped. It's also the single highest-leverage five minutes you'll spend on any experiment. Here's how to do it properly.

Why underpowered tests fail quietly

An underpowered test doesn't fail loudly — it fails by giving you an answer that feels legitimate but isn't. Statistical significance depends on three inputs working together: your baseline conversion rate, the minimum detectable effect (MDE) you're trying to catch, and the traffic volume feeding the test. If any one of those is out of proportion to the others, you get one of two outcomes:

• False negatives: a real 8% lift exists, but your sample was too small to distinguish it from noise, so the test reads "no significant difference" and a winning idea gets shelved.
• False positives: you call a winner at day 9 because the p-value crossed 0.05 for a moment, but the sample was thin enough that the result doesn't replicate post-launch.

Both outcomes cost more than the test itself. A false negative kills a roadmap idea that might've been worth 2-3 points of CVR. A false positive gets built into production code, QA'd, and rolled out — then quietly reversed six months later when someone notices revenue didn't move the way the test promised.

The fix isn't a better statistical test. It's knowing your required sample size before you write a single line of variant code.

The core formula, in plain terms

The full z-test sample size formula for comparing two proportions looks intimidating, but the concept is simple: you're solving for how many observations you need so that a real effect of a given size would show up reliably, given the noise inherent in your baseline conversion rate.

At a 95% significance threshold and 80% statistical power (the industry-standard settings), the formula per variant is approximately:

n = 16 × p × (1 − p) / (MDE × p)²

Where:

• p = baseline conversion rate (as a decimal)
• MDE = minimum detectable effect, expressed as a relative lift (e.g., 0.15 for 15%)
• 16 = a constant derived from the z-scores for 95% confidence and 80% power (roughly (1.96 + 0.84)² × 2)

You'll rarely calculate this by hand in practice — tools like Evan Miller's sample size calculator or AB Test Guide do it instantly — but understanding the mechanics matters because it tells you which lever to pull when your required sample size looks unrealistic.

As a worked example from SplitMetrics' breakdown of the formula: if your baseline conversion rate is 20% and you want to detect a lift to 26% (a 6-point absolute, 30% relative improvement), you need roughly 608 visitors per variant — 1,216 total — to hit significance at 95% confidence with 80% power. That's a large, easy-to-detect effect on a healthy baseline rate, which is why the number stays manageable.

Compare that to a more realistic ecommerce scenario: a 2% baseline conversion rate with a hoped-for 10% relative lift (2.0% → 2.2%). As Mida's traffic guide shows, that combination requires around 60,000 visitors per variation — 120,000 total for a single A/B test. Same statistical rigor, wildly different traffic requirement, purely because the baseline rate is lower and the effect you're chasing is smaller.

The four zones of testing difficulty

Once you internalize how baseline rate and MDE interact, a useful mental model emerges: your monthly traffic volume puts you into one of a few "zones" of testing difficulty. Convertize's framework frames this well:

• Under 10,000 monthly visitors: You're largely restricted to detecting massive effects — think full page redesigns or pricing changes — and even then, tests will run for months. Most micro-copy and button-color tests aren't statistically viable here.
• 10,000–100,000 monthly visitors: The "difficult" zone. You need a conversion rate improvement of at least 9% to reach reliable significance in a reasonable timeframe. This rules out most subtle UI tweaks.
• 100,000–1,000,000 monthly visitors: The "exciting" zone, where you can reliably detect effects between 2% and 9%, depending on exact traffic and baseline rate. This is where most mid-market SaaS and ecommerce CRO programs operate.
• 1,000,000+ monthly visitors: You can chase effects under 2%, which opens up granular testing — copy variants, layout micro-adjustments, personalization rules — that would be statistical noise at lower volumes.

The practical takeaway: know which zone you're in before you build a test roadmap. Teams under 100K monthly visitors who insist on testing subtle variations are setting themselves up for months of inconclusive data. The right move at that traffic level is to test bigger, bolder changes less frequently — not more variants, more often.

Sizing the test to the traffic you actually have

Once you know your zone, work backward from available traffic rather than forward from a wishlist of tests. Three practical levers control your required sample size, and you should always check them in this order:

1. Raise your MDE threshold. As Digital Applied's CRO guide notes, setting MDE too small (1-2%) balloons sample size requirements past what most sites can generate in a quarter. For most ecommerce and SaaS tests, a 10-15% relative MDE is the pragmatic sweet spot — big enough to detect with reasonable traffic, small enough to still be commercially meaningful.
2. Combine low-traffic pages into a single test. If your pricing page and your upgrade-prompt modal serve a similar audience with a similar goal, consider testing the underlying mechanism (e.g., anchoring strategy) across both rather than splitting traffic across two separate low-powered tests.
3. Extend test duration deliberately, not reactively. A benchmark worth keeping in your back pocket: at a 3% baseline CVR targeting a 15% relative lift, Grow Conversions' best-practices guide puts the requirement at roughly 10,000 visitors per variant — a number many mid-traffic sites can hit in 2-4 weeks. Use that as your default planning assumption, then adjust for your own baseline rate.

Run this math using an ICE framework alongside it — Impact, Confidence, Ease — but add a fourth check: Powerable. A test can score high on Impact and Confidence and still be a bad candidate for this sprint if your traffic can't power it within a reasonable window. I've seen roadmaps where the highest-ICE-scored test sat in "running" status for four months because nobody checked whether the page had the traffic to support it.

Duration math: the variable teams forget

Sample size gives you a number of visitors. Duration tells you how many calendar days that translates to, and that's where most timeline estimates go wrong. Divide your required sample size (per variant, times number of variants) by your page's average daily unique visitors, then add a buffer for weekly seasonality — most consumer behavior shifts meaningfully between weekday and weekend traffic, so tests should run in full-week increments, minimum two weeks, to avoid contaminating results with day-of-week effects.

A test that needs 60,000 visitors per variant on a page getting 5,000 daily uniques isn't a two-week test — it's a 24-day minimum, and realistically closer to five weeks once you build in the buffer. If that timeline doesn't fit your roadmap cadence, that's a signal to revisit your MDE or find a higher-traffic page to test the same hypothesis on, not to launch anyway and call it early.

Building sample size checks into your test intake process

The teams that avoid underpowered tests aren't smarter about statistics — they've just made the calculation mandatory before a test gets a launch date. A simple intake gate works well: before any experiment moves from "backlog" to "in development," require a filled-in sample size calculation with baseline rate, target MDE, and projected duration attached to the ticket. If the projected duration exceeds a threshold (say, 6 weeks), the test gets flagged for review — either the MDE gets adjusted, the page gets swapped for a higher-traffic equivalent, or the test gets deprioritized in favor of something powerable this quarter.

This is a five-minute check per test, and it's the difference between a testing program that compounds learnings and one that accumulates inconclusive results nobody trusts.

The takeaway for this sprint

Before your next test goes into development, run the sample size math using your actual baseline conversion rate and a realistic MDE — not the number you hope to see, but the smallest lift that would still be commercially worth shipping. Compare that requirement against your daily traffic to get a real duration estimate, and if it stretches past six weeks, treat that as a signal to widen your MDE or pick a different page — not a reason to launch and hope. Add this single calculation as a required field on your experiment intake template this week, and you'll cut inconclusive tests out of your roadmap before they ever consume a sprint.