The ABWatcher blog

How to Calculate A/B Test Sample Size Without Wasting Traffic

A no-BS guide to sample size math for A/B tests — power analysis, MDE, and the stopping-rule mistakes that quietly wreck test data.

Diego Hernandez

Growth Marketer · · Updated

TL;DR

  • Most tests need 1,000+ visitors and 100+ conversions per variant to hit reliable significance, per Optibase's testing guide.
  • Peeking at results daily and stopping early is the #1 cause of false positives — don't do it.
  • Lever ran a form-length test (short vs. long demo form) detected 13 June 2026, bucketed server-side across five+ fields of difference — exactly the kind of test that needs serious sample size before you trust a winner.
  • Zillow toggled an entire 5-step BuyAbility section on and off, detected 24 May 2026 — a layout change big enough to move the needle fast, but still requires a pre-set sample target.
  • Minimum Detectable Effect (MDE) is the lever most teams ignore — smaller MDE means exponentially more traffic required.
  • Formula from HubSpot: Required Sample Size ÷ Average Weekly Visitors = Minimum Test Duration in weeks.

Why does sample size even matter if the test "looks" significant?

A test can show a 20% lift and still be statistical noise if the sample is too small. Significance without power is a coin flip dressed up as data.

Here's the trap: your dashboard turns green, someone screenshots it for Slack, and the test gets called at day 4 with 340 visitors per arm. That's not a result. That's a rounding error with a confidence interval wide enough to drive a truck through.

Real tests in the wild don't get called that fast. Take AB Tasty, detected running a full server-side hero and brand split on 17 September 2026 — two completely different destinations (wingify.com vs. abtasty.com), different messaging, different CTAs. That's a test with enough structural difference that effect size is probably large. But "probably" isn't a stopping rule. You still need the math.

Takeaway: if you can't state your sample size target before launch, you don't have a test — you have a vibe check.

How do you actually calculate the sample size you need?

You need four inputs: baseline conversion rate, minimum detectable effect, statistical power, and significance level — plug them into a calculator, don't eyeball it.

The four inputs, explained fast:

  • Baseline conversion rate — your control's current conversion rate. Pull 30 days of real data, not a guess.
  • Minimum Detectable Effect (MDE) — the smallest lift you actually care about catching. Chasing a 2% lift costs way more traffic than chasing a 20% lift.
  • Statistical power — usually set at 80%. This is your odds of detecting a real effect if one exists.
  • Significance level (alpha) — usually 95% confidence, meaning a 5% false-positive tolerance.

SplitMetrics breaks down the actual formula with a clean example: control converts at 20%, variant hypothesized at 26% — a 6-point absolute lift. That size of a swing needs meaningfully less traffic than hunting for a 1-point move. Smaller MDE = bigger sample, every time. That relationship is non-negotiable and it's the thing most roadmaps get wrong.

Takeaway: decide your MDE before you decide your test idea. If you can't afford the traffic to detect it, don't run it.

What's a realistic sample size target for most tests?

For most mid-traffic sites, plan for at least 1,000 visitors and 100 conversions per variant as a floor, not a target to brag about.

That's the rule of thumb from Optibase's A/B testing guide — and it holds up in practice. Below 100 conversions per arm, your confidence intervals are so wide that "winning" variants regress to the mean the moment you scale them.

Here's how that floor scales with different baseline conversion rates and target MDEs (approximate, using standard 80% power / 95% confidence):

Baseline CR Target MDE (relative) Approx. sample size per variant
2% 20% lift ~39,000
5% 20% lift ~15,000
10% 15% lift ~11,500
20% 10% lift ~13,000
20% 30% lift ~1,600

Notice the pattern: low baseline conversion rates and small MDEs both blow up your required sample. A pricing page test — like the one Deel ran, detected 17 September 2026, swapping "Book a demo" for "Speak with a trusted expert" and rewriting plan card copy — touches a page with naturally lower traffic and higher stakes per visitor. That combo means longer test durations almost by default. Plan for it instead of getting surprised by it.

Takeaway: before you scope a test on a low-traffic page, run the sample size math first — it might tell you the test isn't worth running yet.

How long should a test actually run?

Run it long enough to hit your calculated sample size, and never shorter than one full business cycle (usually 7-14 days minimum).

HubSpot's formula is dead simple: Required Sample Size ÷ Average Weekly Visitors = Minimum Test Duration in weeks. If you need 15,000 visitors per variant and you're splitting 10,000 weekly visitors 50/50, you're looking at 3 weeks minimum.

Two things trip people up here:

  1. Day-of-week bias. B2B SaaS traffic looks nothing like itself on a Tuesday vs. a Saturday. Cutting a test short means you've only sampled part of your actual buyer behavior.
  2. Novelty effects. A redesigned hero might spike CTR in week one just because it's new, then flatten. Ally ran a hero/banner test detected 17 September 2026 — "More bank for your buck" paired with a WNBA promo vs. "The bank fans bank on" paired with a different seasonal banner. Promo-tied creative like that is especially vulnerable to short-window noise. You need enough duration to see past the initial curiosity click.

Takeaway: set your test duration from the math, then don't touch the "end test" button until you hit it — full stop.

What mistakes cause underpowered or prematurely killed tests?

The three big ones: peeking and stopping early, ignoring segment-level dilution, and testing too many variants on too little traffic.

Mistake 1: Stopping the moment you see a "significant" result. Checking your dashboard daily and calling the test the first time p < 0.05 shows up is the single biggest way to generate false winners. Significance has to be sustained across your full calculated sample, not caught in a lucky window.

Mistake 2: Ignoring that your test might be multivariate in disguise. Lever's detected test (13 June 2026) compared a 4-field short form against a 7-field long form with phone number, job title, company size, and country all added at once. That's not one variable — it's four. If the long form wins, you don't actually know which field drove it. You've spent your whole sample budget and still can't act on the insight. Test one meaningful change at a time, or budget separately for a follow-up isolation test.

Mistake 3: Splitting traffic too many ways. Running a 4-arm test on traffic that barely supports a clean A/B comparison guarantees an underpowered result on every arm. If your total traffic only supports one well-powered comparison, run one.

Mistake 4: Assuming layout toggles are "low risk" and skipping the sample size step entirely. Zillow's BuyAbility section toggle, detected 24 May 2026, is a good example of a test that looks simple — show a section, hide a section — but still sits on a high-traffic, high-stakes page where a false winner gets rolled out to everyone. Simple tests still need real sample size targets. "It's just a visibility toggle" is not a statistical argument.

Takeaway: write your stopping rule and sample size target into the experiment doc before launch — not as a formality, as the thing that keeps you honest when the dashboard starts looking exciting on day 3.

Frequently asked questions

How many conversions do I need per variant for a reliable A/B test?

Aim for at least 100 conversions per variant as an absolute floor, alongside roughly 1,000+ visitors per variant depending on your baseline conversion rate. Below that, confidence intervals are too wide to trust, and a "winning" variant often fails to replicate once you scale it to 100% of traffic.

What's the difference between statistical significance and sample size?

Significance (usually 95% confidence) tells you how likely a result is due to chance. Sample size is what gives that significance calculation enough data to be meaningful. You can hit 95% significance on a tiny, underpowered sample and still be wrong — significance without adequate sample size is misleading, not reliable.

Why did my test show a winner early, then flatten out later?

This is usually novelty effect or early-peek bias. Visitors react differently to a new layout in the first few days, and small early samples amplify random swings. Running the test to your full pre-calculated sample size, across multiple business cycles, filters this noise out before you make a rollout decision.

Can I test a big change like Zillow's section toggle with a smaller sample?

Bigger effect sizes do need smaller samples, which helps with big visual or structural changes. But "it looks big" isn't a substitute for calculating your MDE and running the numbers — even high-traffic pages generate false positives if you stop the moment results look favorable instead of hitting your target sample.

How do I handle sample size when testing multiple changes at once, like Lever's longer form?

Multivariate changes — new fields, new copy, new layout all at once — need you to either treat it as one combined variant (and only claim the combined result, not attribution to any single field) or budget for sequential isolation tests. Trying to attribute a win to one of four changed fields without separate tests is guessing, not analysis.

This sprint's takeaway

Before your next test goes live, write down three numbers: baseline conversion rate, your MDE, and the sample size that math requires. Put the number in the experiment doc. Set a "do not stop before this date" rule and defend it when the dashboard looks tempting on day 3. That's the whole discipline — the hard part is not the math, it's not touching the button early.

Read next

More from the ABWatcher blog

From the glossary

Vocabulary the article uses. See all terms →

  • Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
  • Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
  • Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
  • Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
  • Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
  • Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

See more like this

ABWatcher catches A/B tests like this across 1,000+ brands.

Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.