A/B Test Duration: How Long to Run an Experiment for Real Certainty

Every experimentation program eventually hits the same wall: a test shows a 12% lift on day four, someone screenshots it for the Monday standup, and the pressure to call the test builds faster than the sample size does. The single most common cause of bad experimentation decisions isn't a broken test — it's a test that was read too early, or stopped for the wrong reason entirely.

Duration is not a settled-science question, but it's also not a gut call. It's a function of three inputs you can calculate before you ship a single variant: baseline conversion rate, minimum detectable effect (MDE), and daily traffic. Get those three right, and "how long do I run this" stops being a debate and starts being arithmetic.

Why duration is a math problem, not a calendar guess

Statistical significance depends on sample size, and sample size depends on how big a difference you're trying to detect. This is the part teams skip: they pick a two-week test window because two weeks feels like a reasonable amount of time, then work backward to justify it. That's inverted.

The correct sequence is:

1. Estimate your baseline conversion rate from the last 30-90 days of stable traffic.
2. Decide the smallest lift that would actually matter to the business (your MDE) — 5% relative lift behaves very differently in the math than 20%.
3. Feed both into a sample size calculator to get required visitors per variant.
4. Divide by daily traffic per variant to get calendar days.

As Cro Metrics notes, detecting a 10%-or-larger effect on a moderate-traffic page can require roughly 30,000 visitors per variation — a number that surprises most teams the first time they see it, because it's an order of magnitude higher than what "felt like enough" data looked like in the dashboard. Smaller MDEs require dramatically larger samples: halving the effect size you're trying to detect roughly quadruples the sample size needed, since required sample size scales with the inverse square of the effect.

This is why a checkout page redesign testing for a 15% conversion lift might resolve in 9 days, while a subtle CTA copy tweak chasing a 3% lift on the same page could need 90 days to reach the same confidence level. Same page, same traffic, wildly different timelines — because the effect size assumption is doing all the work.

The traffic-to-time conversion in practice

Once you have required sample size, the conversion to calendar days is simple division, but three practical factors distort it:

• Number of variants. A/B/n tests split traffic further. A test with three variants against a baseline of 30,000-per-arm needs 120,000 total sessions, not 30,000 — a mistake that inflates timeline estimates by 3-4x if missed.
• Weekly cyclicality. B2B SaaS traffic often dips 40-60% on weekends; ecommerce can spike on weekends. Running a test for exactly 7 or 14 days (full week multiples) avoids day-of-week bias contaminating the read.
• Traffic seasonality and promo calendars. A test that straddles a product launch, a pricing change, or a major holiday will absorb variance from those events, not just from your variant. As Elevation B2B points out, appropriate duration accounts for business cycles, not just raw visitor counts.

A rough worksheet that holds up across most SaaS and ecommerce contexts:

Required sample size (per Optimizely/VWO calculator): N
Number of variants (including control): V
Daily unique visitors to test page: D
Required daily visitors per arm: D / V
Estimated days to reach significance: (N × V) / D
Round up to nearest full week: Y/N

If that final number comes out under 7 days, be suspicious — it usually means your MDE assumption is too generous, your traffic estimate is inflated, or you're looking at a page with unusually high intent (like a pricing page for an already-considered purchase) where effect sizes really can be large.

When "statistically significant" isn't done yet

A p-value crossing 0.05 on day 3 is not a green light — it's the single biggest trap in test-reading, commonly called peeking bias. Every time you check a test and make a stop/continue decision based on the current p-value, you inflate your false-positive rate. Checking daily over a two-week test can push your actual false-positive rate well past 30%, even though your dashboard still displays "95% confidence."

Contentful's testing guide frames this plainly: the statistical reliability of a result depends on both sample size and duration together, and stopping early — even at an apparently significant result — risks a decision built on noise rather than signal.

Three guardrails fix this without requiring a stats degree:

• Pre-register your stopping rule. Decide the sample size and/or day count before launch, write it in the test brief, and don't deviate unless there's a legitimate quality issue (broken tracking, a bug in the variant).
• Use sequential testing methods if you need to peek. Tools built on always-valid inference (mSPRT-based platforms, or Bayesian frameworks with proper priors) let you check results continuously without inflating false positives — but only if the platform is actually built for it, not just a standard frequentist dashboard with an early-peek habit layered on top.
• Run a minimum of one full business cycle, typically 7 or 14 days, even if your calculator says you hit sample size in 4. Day-of-week effects on both traffic volume and visitor intent are real, and a Tuesday-heavy sample doesn't generalize to a full week.

The Figr best-practices guide makes a related point worth internalizing: running the calculation upfront turns the test into "a structured scientific inquiry" with a shared definition of done — which matters organizationally as much as statistically, because it removes the ambiguity that lets stakeholders push for an early call when the trend looks favorable.

What ABWatcher sees across live tests

Patterns across the live tests we track daily at high-traffic consumer brands are consistent with the math above: tests on high-intent, low-traffic pages (upgrade flows, annual-plan toggles, add-to-cart on niche SKUs) tend to run 3-5 weeks, while homepage hero and navigation tests on high-traffic sites often resolve in under two weeks because the sample size accumulates faster even though the expected effect size is usually smaller. The duration difference isn't about page importance — it's a direct function of the traffic-times-effect-size equation.

We also consistently see multi-week gaps between a variant going live and a "winner" being declared publicly — evidence that the more disciplined programs are holding to a pre-set duration rather than calling tests on the first favorable read.

Common mistakes that quietly invalidate a duration decision

• Confusing statistical significance with practical significance. A test can reach p<0.05 with an effect size too small to matter to revenue — especially with very high traffic pages where even a 0.5% difference becomes "significant" given enough volume.
• Ignoring novelty effects. New designs sometimes get an initial engagement bump simply because they're different, not better. This is most visible in tests under 5-7 days; extending duration lets novelty decay and reveals the steady-state effect.
• Segment-shopping after the fact. Slicing results by device, geo, or channel until a subgroup looks significant is a form of peeking bias in disguise. Pre-specify any subgroup analysis in the test brief, not after the topline result disappoints.
• Underestimating variance in the metric itself. Revenue-per-visitor and other continuous metrics — as Reform's guide to A/B test metrics notes — often carry more variance than binary conversion metrics, meaning tests optimizing for RPV typically need longer runtimes than tests optimizing for conversion rate alone, even on identical traffic.

Applying an ICE lens to duration tradeoffs

When a stakeholder pushes to end a test early, it helps to reframe the decision using the same Confidence-Impact-Ease structure used for prioritization:

• Confidence: What's the actual false-positive risk of stopping now, given how many times the test has already been peeked at?
• Impact: Is the observed effect size large enough to matter even if it holds — or is it within the range that novelty or noise typically produces?
• Ease/Cost: What's the cost of two more weeks of runtime versus the cost of rolling out a false positive to 100% of traffic?

Most of the time, the two-week cost is smaller than teams assume, and the cost of a reversed rollout — re-engineering, re-communicating to stakeholders, rebuilding trust in the test program — is larger.

Building this into next sprint's test plan

Before launching your next experiment, run the four-step worksheet above for every test in the roadmap, not just the ones that feel uncertain. Attach the calculated duration and required sample size to the test brief itself, and set an explicit rule: no read-outs before that date, and no stop/go decision based on a mid-test p-value unless the platform explicitly supports sequential testing. That one process change — pre-committing to a duration before launch — closes most of the gap between a program that runs tests and a program that runs experiments it can actually trust.