The ABWatcher blog

A/B Testing Statistical Significance: How Long to Run Your Test

A practical, math-backed guide to calculating A/B test duration from baseline rate, traffic, and MDE — plus real examples of tests live on production sites right now.

Diego Hernandez

Growth Marketer · · Updated

TL;DR

  • Minimum sample size for a typical ecommerce test (3% baseline, 10% relative MDE, 80% power) is roughly 28,700 visitors per variant.
  • Most teams that "call it early" do so at day 4-6 — before a single weekly business cycle has completed.
  • Magic Spoon was caught running a hero variant detected 29 May 2026, visible only in Chrome — a sign of a live, actively-monitored split test, not a bug.
  • Deel was mid-test on its pricing page as of 17 Sep 2026, swapping CTA copy and plan card language across variants.
  • Zip had two different sticky-nav CTAs running simultaneously, detected 8 Aug 2026 — "Become a merchant" vs. "Get started."
  • Industry guidance consistently points to 1-2 full business cycles (usually 2 weeks minimum) before reading results — see Wingify's duration breakdown.

Why does test duration matter more than most teams think?

Stopping a test too early inflates your false-positive rate — you'll ship "winners" that are actually noise. Duration isn't a formality. It's the difference between a real lift and a coin flip you convinced yourself was a strategy.

Here's the uncomfortable math: if you peek at your results every day and stop the moment you see a "significant" result, your actual false-positive rate isn't 5%. It can climb to 20-30%, depending on how often you peek. That's not statistics — that's p-hacking with a dashboard.

This is the single most common CRO mistake we see. Not bad hypotheses. Not ugly variants. Just impatience.

How do you actually calculate the sample size you need?

You need four inputs: baseline conversion rate, minimum detectable effect (MDE), statistical power, and significance threshold — then run them through a sample size calculator.

The formula itself isn't the hard part. Any calculator will spit out a number. The hard part is being honest about your inputs before you see data, not after.

Here's what the four inputs actually mean:

  • Baseline conversion rate — your current rate for the metric you're testing (e.g., 3% checkout completion).
  • Minimum detectable effect (MDE) — the smallest lift you actually care about detecting. Smaller MDE = way more traffic needed.
  • Statistical power — typically set at 80%. This is your chance of detecting a real effect if one exists.
  • Significance level — typically 95% confidence (alpha = 0.05).

Here's how sample size scales with MDE, holding baseline conversion at 3% and power at 80%:

Minimum Detectable Effect (relative) Required Sample Size (per variant) Approx. Days at 2,000 visitors/day/variant
5% ~111,000 55 days
10% ~28,700 14 days
20% ~7,400 4 days
30% ~3,400 2 days

Notice the pattern: chasing small lifts costs you weeks. If your traffic can't realistically detect a 5% lift in a reasonable window, don't design a test around hoping for one. Design for a bolder change or accept a longer runway.

Why do short tests produce false winners?

Short tests catch noise, not signal, because early data is dominated by random variance and day-of-week effects. A test that looks like a 15% lift on day 2 regularly collapses to flat by day 14.

This is why you see so many "we saw a 40% lift!" screenshots on LinkedIn that never get a follow-up post three weeks later. The lift didn't survive contact with a full data cycle.

Two specific failure modes:

1. Novelty effects. Returning visitors react to change itself, not the specific change. A new hero section gets extra clicks simply because it's new. That effect fades. If you stop at day 3, you've measured novelty, not preference.

2. Day-of-week skew. B2B traffic converts differently on Tuesday than Saturday. Ecommerce traffic on payday weekends behaves differently than mid-month. If your test window doesn't span at least one full weekly cycle — ideally two — you're letting calendar noise masquerade as a treatment effect.

Contentful's guide to A/B test duration makes the same point: too short and you get unreliable results, too long and your test stops reflecting current reality (seasonality, promotions, algorithm changes all creep in).

What does a properly-run test actually look like in the wild?

Live tests on real production sites show the mechanics we're describing — variant isolation, controlled rollout, and copy or layout changes tested in parallel rather than sequentially.

Take Magic Spoon, detected 29 May 2026. The Chrome visit showed a completely different hero treatment — a "Testing Testing Testing" placeholder panel with different nutritional callouts (No Grains, No Gluten, No Glyphosates vs. 12g Protein, 3g Net Carbs, 6g Coconut Oil) — while Safari and Firefox rendered the standard hero. That's textbook controlled variant delivery: segment by session, not by browser preference, and keep the control stable while you measure the new concept.

Deel's pricing page test, detected 17 Sep 2026, is a good example of testing message framing rather than just visuals. The variant swapped "Book a demo" for "Speak with a trusted expert," cut the Bundle tab entirely, and rewrote plan card copy from blunt feature labels ("Hire employees") to longer benefit-style language ("Hire full-time employees"). That's a hypothesis about trust and clarity, not just a button color change — the kind of test that needs real sample size because the expected lift is probably modest, not explosive.

Zip's sticky nav test, detected 8 Aug 2026, ran two CTA treatments simultaneously: a lime-green "Become a merchant" button against an outlined "Contact sales," versus a solid purple "Get started" paired with "Contact sales" elsewhere in the rotation. CTA color and label tests are classic high-traffic, low-MDE-tolerance tests — you need volume to separate a real preference from noise, because color swaps rarely move the needle more than single-digit percentages.

And Mailmodo, detected 30 Jun 2026, shows an entire layout variant — a shorter page with just a hero and one feature section — running against a much longer page with testimonials and extra feature blocks, isolated specifically to Safari while Chrome, Firefox, and Edge saw the long version. Structural tests like this (short page vs. long page) typically produce bigger effect sizes, which means they can resolve faster than micro-copy tests — but only if the team lets the full cycle play out instead of calling it on day one.

When do you actually have enough data to make a call?

You have enough data when you've hit your pre-calculated sample size AND covered at least one full business cycle — not when the dashboard first shows green.

Checklist before you ship a "winner":

  1. Did you hit your pre-registered sample size? Not "did the tool say significant" — did you hit the number you calculated before the test started.
  2. Did the test run at least one full week, ideally two? This captures weekday/weekend behavior and typical traffic mix.
  3. Is the result stable, not just momentarily crossing the threshold? Watch for significance flickering in and out over several days — that's a sign you're still inside the noise band.
  4. Did you avoid sequential peeking-and-stopping? If you checked daily and stopped the moment you crossed 95%, your real confidence is lower than it looks. Use a pre-registered duration or a sequential testing method built for repeated looks.
  5. Does the result make directional sense with your hypothesis? A surprising result isn't automatically wrong, but it deserves a second look before you roll it to 100%.

Appinio's testing framework frames this well: define duration upfront, then monitor significance as a checkpoint — not as a stopping trigger you invented after the fact.

Common questions

How long should an A/B test run at minimum?

Most practitioners recommend a minimum of one to two full business cycles, typically 2 weeks, regardless of how fast you hit statistical significance. This accounts for day-of-week variance, novelty effects, and ensures the sample size reflects real traffic patterns, not a lucky few days.

What sample size do I need for 95% confidence?

It depends entirely on your baseline conversion rate and minimum detectable effect. At a 3% baseline rate and 10% relative MDE with 80% power, you need roughly 28,700 visitors per variant. Smaller MDEs require dramatically larger samples — a 5% MDE can require over 100,000 visitors per variant.

Why did my test show significance on day 3 then disappear by day 10?

This is classic early-peek noise combined with novelty effect. Small samples have wide variance, so random swings can cross the significance threshold temporarily. As more data arrives, the estimate stabilizes toward the true (often smaller) effect. Pre-registering your sample size prevents this false alarm.

Can I stop a test early if the result looks really strong?

Only if you're using a sequential testing method designed for repeated peeking (like sequential probability ratio tests). Standard fixed-horizon significance testing assumes you check once, at the end. Stopping early on a standard test inflates your false-positive rate well beyond the stated 5%.

Does a bigger visual change need a shorter test than a copy change?

Generally, yes. Bigger changes (full layout overhauls, like the one detected at Mailmodo) tend to produce larger effect sizes, which need less data to detect. Subtle copy or color tweaks (like the CTA swap at Zip) usually have smaller true effects and need more traffic and time to confirm.

Your next sprint

Before you launch your next test, write down your baseline rate, your MDE, and your target sample size — on paper, before you see a single data point. Then commit to a minimum two-week runway regardless of what the dashboard says on day 4. That one habit kills more false positives than any tool you'll buy this quarter.

Read next

More from the ABWatcher blog

From the glossary

Vocabulary the article uses. See all terms →

  • Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
  • Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
  • Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
  • Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
  • Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
  • Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

See more like this

ABWatcher catches A/B tests like this across 1,000+ brands.

Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.