Sample Size vs. Run Time: How to Choose for A/B Tests
Sample size and run time aren't competing strategies — they're two inputs to the same equation. Here's the decision framework, with real test patterns from live brand experiments.
Jordan Reeves
Founder & Operator · · Updated
Every new tester eventually asks the same wrong question: "Should I optimize for sample size or run time?" That framing sends teams down a rabbit hole — some crank up traffic allocation to hit a number fast, others just let tests ride for "a few weeks" and hope. Both instincts miss the point. Sample size and run time are the same variable viewed from two angles, and treating them as a choice is how teams end up calling tests on Tuesday that would have flipped by Friday.
TL;DR
- A valid test typically needs ~30,000 visitors per variant and 3,000+ conversions per variant as a rough floor (GuessTheTest).
- Stopping early on a "trending positive" result is the single most common cause of false positives in A/B testing (Intempt, 2026).
- Most eCommerce tests should target a 10-15% relative minimum detectable effect (MDE) — smaller MDEs balloon required sample size (Digital Applied).
- Low-traffic sites can still test validly by targeting bigger swings (bold redesigns, not button-color tweaks) rather than running forever.
- ABWatcher detected five live hero, pricing, and copy tests — at SignNow, Disney+, Lyft, Ally, and Buffer — between 8 Jul 2026 and 8 Sep 2026, each shaped by this exact tradeoff.
- Full test duration should always cover at least one full business cycle (commonly 1-2 weeks minimum) regardless of how fast sample size accumulates.
What's the actual relationship between sample size and run time?
Run time is just sample size divided by your daily traffic — they're not two levers, they're one lever measured in two units. If you know your required sample size per variant (say, 30,000) and your daily unique visitors hitting that page (say, 3,000 split across two variants, so 1,500 per variant per day), run time is simple division: 20 days. High-traffic teams hit the sample number in days; low-traffic teams hit it in months. Neither group gets to skip the math — they just experience it differently. The mistake is picking a strategy ("we'll run it for traffic" or "we'll run it for two weeks") before doing that division at all.
When should you optimize for sample size over duration?
Optimize for sample size when you have enough daily traffic that reaching statistical power takes days, not weeks — and when your business cycle is short enough that duration isn't adding information. A homepage hero test on a high-traffic site is the clean case. ABWatcher detected SignNow running a hero A/B test as of 8 Aug 2026, splitting visitors between a single-CTA "Get your first document signed" hero and a two-column "Send my document for signature" / "Sign my own document" layout with an added client-logo strip (Walmart, ExxonMobil, Apple, Comcast, Facebook, FedEx). That's exactly the kind of high-intent, high-traffic landing surface where a team can afford to chase sample size aggressively — the daily volume is there, and the test doesn't need to survive a monthly billing cycle or seasonal swing to be meaningful.
Similarly, Lyft was caught testing subheadline copy in its "Choose how to ride" section as of 8 Jul 2026 — "Find a ride that suits your mood and budget" versus "Hit the road, or the rails. Find a ride that fits your style and budget" — within the same browser engine, ruling out rendering artifacts. Copy tests on high-traffic consumer surfaces like this are prime sample-size candidates: the daily visit volume lets you reach significance in days, and there's no reason to extend runtime just for duration's sake once you've hit your number and covered at least one full week of day-of-week variation.
When does extending run time matter more than hitting a traffic number?
Extend run time when your daily traffic is modest, your conversion event is a longer-cycle decision, or your buyer behavior varies meaningfully by day of week or pay cycle. This is where low-traffic and B2B teams live, and it's also where premature calls do the most damage. Ally was detected running a hero test as of 8 Sep 2026 — one variant with the headline "More bank for your buck." and a lifestyle-brand banner, the other with "The bank fans bank on." and a WNBA sponsorship banner — consistent across browsers and visits. Banking decisions aren't impulse clicks; a visitor might see this hero today and convert two weeks later after comparing rates elsewhere. A test like this needs enough calendar time to capture that full decision window, not just enough raw hits to clear a significance threshold.
The same logic applies to Disney+, detected 20 Jul 2026 running a pricing-page layout test — a card-based "Choose your plan" layout with SELECT buttons versus a text/checkmark comparison table, split by cookie bucket rather than browser engine. Pricing decisions often involve a household discussion, a comparison to a competing service, or a wait for a promo window. Hitting your sample-size number on day 4 doesn't mean the test is done if your buyers typically take 10 days to decide — you'd be measuring the impulsive minority, not your real population.
How do you avoid calling a test before it's actually finished?
Set your sample size and your minimum duration before launch, and don't look at results until both thresholds are met. The most common failure mode in the industry isn't bad math — it's peeking. Teams check results daily, see a trend forming, and ship the "winning" variant before the test has run long enough or gathered enough conversions, which produces false positives at a high rate because early-stage data is statistically noisy (Intempt, 2026). Buffer was detected running a hero subheadline test as of 26 Aug 2026 — "Connected to every platform and tool you use." versus "Works with every platform you post to, and plugs into your favorite tools." — split consistently across both Chrome and Firefox, which rules out a rendering fluke and signals a real, deliberately bucketed test. That kind of cross-browser consistency is also your sanity check: if a "result" only shows up in one browser engine, you're looking at a rendering artifact, not a test outcome, no matter how clean the numbers look.
Here's a simple before-you-launch checklist that prevents most premature calls:
| Pre-launch check | Why it matters |
|---|---|
| Calculate sample size per variant before launch | Avoids retroactively justifying an early stop |
| Set minimum duration (≥1 full business cycle) | Captures day-of-week and pay-cycle variance |
| Pick MDE of 10-15% relative lift for most tests | Smaller MDEs need exponentially larger samples (Digital Applied) |
| Confirm variant assignment survives browser/engine checks | Rules out rendering artifacts vs. real cookie-bucketed splits |
| Define the stopping rule in writing, before data comes in | Removes the temptation to peek and stop early |
What should low-traffic teams do differently?
Low-traffic teams should test bigger, bolder changes instead of trying to out-wait their traffic ceiling. A small-traffic site chasing a 2% lift on a button color will need months to reach 30,000 visitors per variant with enough conversions to resolve the difference — that's often not a realistic roadmap. The smarter move, as the sample-size math implies, is that smaller populations can still detect large differences confidently; it's only small differences that require huge samples (Concord USA). So instead of a subtle copy tweak, test a full hero redesign, a completely different pricing layout, or a new value proposition — changes big enough that if they work, the lift is large enough to detect with the traffic you actually have. As a reference point, general guidance puts a valid test floor around 30,000 visitors and 3,000 conversions per variant (GuessTheTest; Cro Metrics) — if your traffic can't get there in a reasonable window even with a bold test, that's a signal to test qualitatively first (user interviews, five-second tests) before committing engineering time to a live split.
Common questions
Is a bigger sample size always better than a longer run time?
No — they're not substitutes for each other. You need both the statistical sample size and enough calendar duration to capture realistic buying-cycle and day-of-week variance. A test can hit its sample-size number in three days and still be invalid if those three days don't represent a full weekly cycle of your actual traffic mix.
How long should a minimum viable A/B test run?
Most practitioners recommend at least one full week, and ideally one full business or billing cycle, regardless of how quickly you hit your sample-size target. This controls for day-of-week effects, payday timing, and weekly content or marketing pushes that can skew a short window's results.
What's a reasonable minimum detectable effect (MDE) to start with?
For most eCommerce and SaaS tests, a 10-15% relative MDE is a practical starting point — tight enough to catch meaningful wins, loose enough to keep sample size requirements realistic (Digital Applied). Going below 5% MDE usually requires traffic volumes only large consumer platforms have.
How do I know if an A/B test I'm seeing live is real or just a rendering bug?
Check whether the variation holds consistently within the same browser engine across multiple visits — if Chrome always shows version A and Firefox always shows version B, that's likely a rendering artifact, not a bucketed test. Real tests, like the ones ABWatcher detected at Lyft and Buffer, show variation within the same engine.
What should a small team with limited traffic test first?
Test big, structurally different variants — a new hero narrative, a different pricing table format, or a completely different CTA flow — rather than micro-tweaks. Bold changes produce detectable effects with smaller samples, which matters when your daily traffic can't support a months-long test for a 2% lift.
What I would actually do next
Before your next sprint planning call, pull your actual daily traffic for the page you want to test, run it through a sample size calculator with a realistic 10-15% MDE, and divide to get your real run-time estimate — then write both numbers down before anyone touches the experiment tool. If the math says your test needs 11 weeks at current traffic, don't shrink the MDE to make the number look better; shrink the ambition of a different test instead, or go bigger on the variant itself. The teams running clean, legible tests — the kind ABWatcher catches holding steady across dozens of visits and multiple browsers — aren't the ones with the most traffic. They're the ones who did this arithmetic before they shipped the experiment, not after they started squinting at day-three results.
Read next
More from the ABWatcher blog
Nov 22, 2026
How to Calculate Confidence Intervals for Conversion Rate Lift
A practical guide to computing confidence intervals for A/B test conversion lift — frequentist and Bayesian methods, worked examples, and how to report ranges instead of fake certainty.
Nov 21, 2026
Why Your Conversion Rate Varies by Traffic Source (And What to Do About It)
A diagnostic guide to conversion rate variance across organic, paid, and referral traffic — how to separate real signal from source-quality noise before you spin up a new test.
Nov 20, 2026
Why Your A/B Test Winners Don't Replicate Across Traffic Sources
A test winner that holds for blended traffic often fails for a specific channel. Here's why, with real examples, and how to validate before you scale.
Nov 19, 2026
Session Recording for CRO: What to Watch When Users Bounce
A practical, intro-level guide to reading session replays for hidden friction and high-intent signals — plus how to turn what you see into testable hypotheses.
Nov 18, 2026
How to Build a Winning Landing Page Without A/B Testing
No traffic for statistical significance? Here's a design-led framework for building high-converting landing pages using proven patterns, not guesswork.
Nov 17, 2026
How to Find Your True Conversion Bottleneck (Not Just the Waterfall)
A step-by-step method for pinpointing the real cause of funnel drop-off using session recordings, heatmaps, and field-level friction data — before you waste a test cycle.
From the glossary
Vocabulary the article uses. See all terms →
- Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
- Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
- Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
- Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
- Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
- Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See more like this
ABWatcher catches A/B tests like this across 1,000+ brands.
Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.