The ABWatcher blog

How to Calculate Minimum Detectable Effect for A/B Tests

A practitioner's guide to setting MDE before you launch a test — the math, the tradeoffs with sample size and duration, and real test patterns from live A/B programs.

Sam Lee

Data Analyst · · Updated

TL;DR

  • MDE formula: MDE = (Z_α/2 + Z_β) × √(2 × p × (1−p) / n) — set before the test runs, not after.
  • Detecting a 15% relative lift on a 3% baseline (3.0% → 3.45%) needs roughly 23,000 visitors per arm, ~46,000 total, at 80% power / 95% significance.
  • Zillow ran a layout A/B test on its BuyAbility 5-step section, detected 24 May 2026 — the kind of structural change that needs a large MDE to justify shipping.
  • Deel tested pricing page CTA and plan-copy variants, detected 17 Sep 2026 — copy tests like this usually chase smaller, harder-to-detect lifts.
  • AB Tasty ran a full server-side redirect split between two branded destinations, detected 17 Sep 2026 — an extreme-variance test design that changes the MDE math entirely.
  • Lowering your MDE from 10% to 5% relative lift roughly quadruples the required sample size — plan traffic accordingly.

What is minimum detectable effect and why does it gate your test design?

Minimum detectable effect is the smallest true lift your test is statistically powered to catch, and it has to be fixed before you collect a single visitor.

MDE isn't a reporting metric — it's a design input, like baseline conversion rate or significance threshold. If you don't set it deliberately, your test's duration and traffic allocation set it for you implicitly, and often badly. As Kissmetrics explains, the formula is:

MDE = (Z_α/2 + Z_β) × √(2 × p × (1 - p) / n)

Where p is your baseline conversion rate, n is sample size per variant, Z_α/2 is 1.96 for 95% significance, and Z_β is 0.84 for 80% power. The practical implication: if your calculated MDE is 10% relative and your true effect is 3%, the test will report "not significant" — not because there's no effect, but because you never built the test to see it. That's not a null result, it's an underpowered one, and the two get confused constantly in postmortems.

This matters most on structural changes. Zillow's BuyAbility layout test, detected 24 May 2026, toggled an entire 5-step process section (Calculate BuyAbility → Get pre-approved → Make an offer → Apply for a mortgage → Close on your home) on and off across sessions. A change that visible is usually hypothesized to move the needle by double-digit percentages — which is good, because large structural tests are exactly where teams can afford a larger MDE and still finish in a reasonable window.

How do you calculate sample size once you've picked an MDE?

Sample size per variant scales with the inverse square of your MDE, so halving your MDE roughly quadruples your required traffic.

Using the standard two-proportion formula referenced in Redclawey's 2026 experimentation guide:

n = (Z_α/2 + Z_β)² × 2 × p × (1 - p) / (p1 - p2)²

Concretely, per the Medium walkthrough on sample size calculation: a 3% baseline add-to-cart rate, targeting a 15% relative lift (3.0% → 3.45%), at 80% power and 95% significance, requires about 23,000 visitors per arm — 46,000 total for a two-variant split. Run the same baseline but target a 5% relative lift instead, and you're looking at roughly 9-10x that traffic, often 400,000+ visitors. This is why most teams quietly inflate their MDE target under traffic pressure, then wonder why they keep failing to detect lifts their intuition says are real.

Here's how required sample size moves with target MDE at a fixed 3% baseline, 80% power, 95% significance:

Target relative MDE Approx. visitors per arm Approx. total traffic
30% ~4,200 ~8,400
20% ~9,500 ~19,000
15% ~23,000 ~46,000
10% ~52,000 ~104,000
5% ~210,000 ~420,000

The curve isn't linear — it's quadratic — and that's the single fact stakeholders most often miss when they ask "can we just detect a smaller win?"

How does traffic volume constrain which MDE you can realistically chase?

Your available weekly traffic and baseline conversion rate set a floor on the MDE you can detect in a given test window, so work backward from traffic, not forward from ambition.

Wingify's framework flips the standard calculation: instead of picking an MDE and solving for sample size, estimate your available traffic in weekly blocks, plug in your historical conversion rate, and solve for the minimum MDE achievable in that window. With 50,000 weekly visitors at a 3% baseline, a two-week test caps you at a certain MDE floor — run it longer, and the floor drops, but diminishing returns set in past 3-4 weeks for most SaaS funnels due to novelty effects and seasonality.

This is where the AB Tasty detection is instructive. AB Tasty's hero test, detected 17 Sep 2026, didn't tweak a button — it server-side split traffic between two entirely different destinations and brand systems (wingify.com's pastel "Every visitor is different" messaging vs. abtasty.com's neon "Push the limits of experience optimization with AI-powered conversions" hero). Tests with that much variance between arms typically see large effect sizes, which means they can be adequately powered on far less traffic than a subtle copy tweak. If you're testing something that dramatic, don't borrow a sample-size calculation built for a 2% button-color lift — the variance structure is different and your MDE target should reflect that.

Compare that to Deel's pricing page test, detected 17 Sep 2026: a different hero CTA ("Speak with a trusted expert" vs. "Book a demo"), a removed "Bundle" tab, and rewritten plan card copy. These are incremental, funnel-stage changes — the kind that plausibly move conversion by 3-8%, not 30%. That calls for a much larger sample and longer runtime, and if Deel's pricing page doesn't clear six figures in monthly sessions, a 5% MDE target may simply not be achievable in a quarter. The honest move in that situation is either to widen the MDE target (and accept you'll only catch bigger wins) or redesign the test to measure a higher-traffic upstream event as a proxy.

Why do copy and personalization tests need a different MDE strategy than layout tests?

Personalization and copy tests usually chase smaller per-user effects but can be evaluated on engagement proxies with far higher event volume than conversion.

Lattice's AI Agent widget test, detected 13 Jun 2026, varied the chat prompt copy by browser/session — generic "AI Agent" label in some visits versus a personalized pitch ("I'm Lattice from Lattice… curious how building dream teams could transform your team's performance?") in others. A test like this is rarely going to move bottom-funnel conversion by double digits on its own; the realistic effect size is probably in the 2-6% range on an engagement metric like widget-open rate or chat-initiation rate, not demo bookings.

The fix is to measure the test against a higher-volume proxy metric. If demo bookings are too sparse to power a 5% MDE, but widget interactions are 10-20x more frequent, you can detect a real, smaller effect on the proxy with a fraction of the traffic — then validate directionally against the downstream conversion metric once you have enough volume to trust it. This two-tier approach is standard in high-traffic SaaS testing programs and avoids the trap of either underpowering the real metric or inflating your MDE until the test is meaningless.

What else do teams get wrong when setting MDE?

Three recurring mistakes: ignoring segment-level power, treating "not significant" as "no effect," and not accounting for novelty decay.

First, an aggregate test can be well-powered overall while being badly underpowered on the segments stakeholders actually care about. If your test is powered for a 10% MDE at n=50,000 total, but marketing wants to know the effect on paid-traffic visitors who are 20% of that sample, your segment-level MDE just roughly doubled — you'd need a 20%+ true effect in that segment to detect it reliably. Always report segment sample sizes alongside the topline, especially for anything feeding a go/no-go decision.

Second, per DigitalApplied's CRO guide, 95% significance alone tells you nothing about whether your test could have detected the effect size that actually existed. A p-value of 0.42 on a test powered only to 20% MDE is uninformative about a true 6% lift — it's not evidence of "no difference," it's evidence of "we didn't look hard enough."

Third, flag novelty and selection effects before you size the test, not after. Ally's hero test, detected 17 Sep 2026, ran two full headline/banner pairings ("More bank for your buck" + WNBA promo vs. "The bank fans bank on" + a separate seasonal banner) concurrently across the same browser engines. Seasonal or promo-tied creative like this is especially vulnerable to novelty effects and calendar confounds — a lift measured in week one of a promo tie-in may regress once the novelty or seasonal relevance fades. If your MDE calculation assumes a stable baseline conversion rate across the test window, verify that assumption holds before you trust the final number, particularly for time-bound campaign creative.

Common questions

What's a reasonable MDE to target for a typical SaaS pricing page test?

Most SaaS pricing tests target 10-20% relative lift as their MDE, since smaller effects require traffic volumes (100,000+ visitors) that most pricing pages don't generate in a reasonable test window. If your pricing page gets under 20,000 monthly sessions, targeting below a 15% MDE usually isn't practical without running the test for months.

How long should I run a test to hit my target MDE?

Divide your required total sample size (from the MDE formula) by your average weekly traffic, then round up to the nearest full week to avoid day-of-week bias. Most practitioners also enforce a 2-week minimum regardless of math, to capture a full weekly cycle and reduce novelty-effect distortion in the early days of a test.

Can I lower my MDE mid-test if results look promising?

No — recalculating or peeking at MDE mid-test without correction inflates your false positive rate well above 5%, a classic "peeking problem." If you need flexibility, use a sequential testing method or alpha-spending function designed for interim looks, not an ad hoc recalculation.

Does a bigger visible change (like a full redesign) always mean I can use a larger MDE?

Usually yes, but verify it with a pilot rather than assuming it. Large structural or branding changes — like the full destination split AB Tasty ran between wingify.com and abtasty.com — often produce bigger effect sizes, which justifies a larger MDE and smaller sample, but a pilot week of data will tell you faster than intuition alone.

What sample size do I need to detect a 15% relative lift from a 3% baseline?

Roughly 23,000 visitors per variant, or about 46,000 total for a standard two-arm test, at 80% power and 95% significance. This comes directly from the two-proportion sample size formula and is a useful benchmark to sanity-check any calculator output.

The takeaway for your next sprint

Before you greenlight your next test, write down three numbers on the ticket: baseline conversion rate, target MDE, and required sample size — then check that number against your actual weekly traffic. If the math says 12 weeks and your roadmap says 2, don't quietly ship an underpowered test and call a p=0.08 result "directionally positive." Either widen the MDE target to match your real traffic ceiling, pick a higher-volume proxy metric, or kill the test idea and spend the slot on something you can actually power.

Read next

More from the ABWatcher blog

From the glossary

Vocabulary the article uses. See all terms →

  • Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
  • Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
  • Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
  • Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
  • Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
  • Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

See more like this

ABWatcher catches A/B tests like this across 1,000+ brands.

Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.