The ABWatcher blog

How to Calculate Confidence Intervals for Conversion Rate Lift

A practical guide to computing confidence intervals for A/B test conversion lift — frequentist and Bayesian methods, worked examples, and how to report ranges instead of fake certainty.

Sam Lee

Data Analyst · · Updated

TL;DR

  • A 95% confidence interval tells you the range of plausible true lift — not a single point estimate like "+12%."
  • On a test with 2,000 visitors per arm and a baseline CR of 3%, the margin of error alone can be ±1.5 percentage points — wider than the "win."
  • ABWatcher detected five live layout/hero/form tests on 24 May, 17 Sep, and 21 Nov 2026 at Zillow, AB Tasty, Snake River Farms, Ally, and Magic Spoon.
  • Bayesian credible intervals answer "what's the probability variant B beats A," which stakeholders intuitively (if wrongly) assume p-values already tell them.
  • A ratio-based uncertainty interval (log-normal method) is more robust for conversion rate comparisons than a naive difference-in-proportions interval — see the Harlan Harris method.
  • Never ship a "win" off a single variant without reporting the interval width and the minimum sample size that would make it trustworthy.

Why do confidence intervals matter more than p-values in CRO?

A p-value tells you whether to reject the null; a confidence interval tells you the size of the effect you're actually shipping. Growth teams don't make roadmap decisions off a binary "significant/not significant" flag — they make decisions off magnitude. A test that hits p < 0.05 with a point estimate of +8% lift but a 95% CI of [+0.4%, +15.6%] is a fundamentally different business case than one with a CI of [+6.9%, +9.1%]. Same p-value, wildly different confidence in the number you'd put in a revenue model.

This matters because CRO benchmarks get cited as if they're precise. One guide notes that hitting 95% significance alone isn't sufficient — you also need adequate sample size to detect your target effect size reliably, via Minimum Detectable Effect (MDE) calculations done before you launch, not after. A test that's underpowered for its true effect size will produce wide, unstable intervals even when it clears the p < 0.05 bar.

How do you calculate a frequentist confidence interval for conversion rate lift?

Use the Wald or log-ratio method on your conversion counts, not just the raw percentage difference. The naive approach — subtracting two proportions and bolting a standard error onto the difference — breaks down when conversion rates are low (under ~5%) or sample sizes are uneven between arms, which is common in real-world tests with imperfect traffic splits.

A more robust approach, from Harlan Harris's write-up on communicating A/B results, models the ratio of conversion rates on a log scale:

cr_ratio <- function(a_ct, b_ct, a_conv, b_conv,
                      percentiles=c(0.025, .5, 0.975), as_percents = TRUE) {
  a_prop = a_conv / a_ct
  b_prop = b_conv / b_ct
  m = log(b_prop / a_prop)
  v = ((1 / a_prop) - 1) / a_ct + ((1 / b_prop) - 1) / b_ct
  ret = exp(qnorm(percentiles, mean=m, sd=sqrt(v)))
  if (as_percents) ret <- 100 * (ret - 1)
  names(ret) = percentiles
  ret
}

Worked example: control gets 10,000 visits, 300 conversions (3.0% CR). Variant gets 10,000 visits, 345 conversions (3.45% CR) — a 15% relative lift. Plugging into the formula above yields roughly a 95% CI of [-1.2%, +33%] relative lift. The point estimate looks exciting; the interval tells you it's also consistent with a near-zero or even slightly negative true effect. That's the gap between "we found a winner" and "we found a plausible winner that needs more data or a replication run."

Rule of thumb for required sample size: to detect a true relative lift of 10% on a 3% baseline CR at 80% power and α = 0.05, you need roughly 15,000–17,000 visitors per arm. Most teams stop tests at a fraction of that and then act surprised their CI is as wide as the effect itself.

How do Bayesian credible intervals differ from frequentist CIs?

A Bayesian credible interval gives you a direct probability statement — "there's a 92% chance variant B's true conversion rate exceeds A's" — which is what stakeholders think a p-value means anyway. Frequentist CIs technically say "if we repeated this experiment infinitely, 95% of such intervals would contain the true value," which is a correct but unintuitive statement that gets mangled in every slide deck that's ever existed.

Bayesian A/B testing models each arm's conversion rate as a Beta distribution (Beta(1,1) prior, updated with observed successes/failures), then simulates or integrates to get:

  • P(B > A) — probability variant B is actually better
  • Expected loss — how much you'd lose on average if you picked wrong
  • A credible interval on the lift itself, e.g., 94% CI of [2%, 19%] relative lift

The practical advantage: Bayesian methods let you make a "ship it" call even before you've hit a fixed sample size, because you're reasoning about the posterior directly rather than waiting for a pre-registered n. The risk: without discipline, teams peek constantly and stop as soon as the posterior looks good, which reintroduces the same optional-stopping bias that invalidates naive frequentist peeking. Pick one framework, pre-register your decision rule, and don't switch frameworks mid-test because the first one didn't give you the answer you wanted.

What does this look like in live tests ABWatcher has detected?

Real tests in the wild are rarely simple headline-vs-headline swaps — they're whole-section toggles and even full redirect splits, which changes how you'd even define your conversion denominator. On 24 May 2026, ABWatcher detected an Optimizely-style experiment at Zillow toggling the entire 5-step "BuyAbility" process section (Calculate BuyAbility → Get pre-approved → Make an offer → Apply for a mortgage → Close on your home) on and off across visits — present in visits 1 and 3, absent in visit 2. That's a structural layout test, and the CI you'd want isn't just "did CR move" but "did it move differently for users who scrolled past the fold," since the section sits mid-page.

On 17 Sep 2026, we caught something more extreme at AB Tasty: a server-side split sending visitors to two entirely different domains — wingify.com with pastel branding and "Every visitor is different" messaging, versus abtasty.com itself with neon branding and "Push the limits of experience optimization with AI-powered conversions." When your "variants" are different domains entirely, your confidence interval needs to account for domain-level confounds (load time, brand trust, SEO history) stacking on top of the creative difference — the effect size you measure is a bundle, not a clean A/B delta.

On 21 Nov 2026, two more: Snake River Farms ran a form test splitting a "$25 OFF sign up for access" email capture plus a below-fold popup, confirmed across matching browser engines (Chrome visit 2 has it, visit 5 doesn't; Firefox visit 7 has it, visit 3 doesn't) — ruling out rendering artifacts as the explanation. Same day, Ally tested a complete hero swap: light-purple "It's time to branch out." versus dark-purple "More bank for your buck.," each with different CTA copy. And Magic Spoon tested "Your Favorite Treat, Reimagined with Protein." against "Childhood Classics. Grown-Up Ingredients." with entirely different product photography.

Here's how these five detections break down by surface and what CI approach fits each:

Company Surface tested Date detected CI consideration
Zillow Mid-page section toggle 24 May 2026 Needs scroll-depth-segmented CI, not just overall CR
AB Tasty Full domain/hero redirect 17 Sep 2026 Confounded effect — widen interval mentally for domain-level noise
Snake River Farms Form + popup presence 21 Nov 2026 Browser-engine cross-check strengthens causal read
Ally Full hero (copy + image + CTA) 21 Nov 2026 Multi-element bundle — can't attribute lift to one component
Magic Spoon Hero headline + image 21 Nov 2026 Classic single-variable-ish test, cleanest for CI math

The lesson: the cleaner the test design (Magic Spoon's hero-only swap), the more defensible a tight confidence interval becomes. The messier the design (AB Tasty's full redirect, Ally's multi-element hero), the more you should widen your stated uncertainty and caveat that the CI reflects a bundle of changes, not an isolated causal driver.

How do you communicate confidence intervals to non-technical stakeholders?

Translate the interval into dollars and a plain-English range, not a Greek-letter formula. Instead of "the 95% CI for lift is [1.2%, 14.8%]," say: "We're 95% confident the true improvement is somewhere between 1 extra conversion per 1,000 visitors and 15 extra — most likely around 7." If your pricing page does $50M ARR and converts at 3%, a point estimate of +4% lift sounds like roughly $2M annualized, but if your CI spans [+0.5%, +8%], the honest range is $250K to $4M — still worth shipping, but not a number you'd put in a board deck without the caveat.

Three communication habits that hold up under scrutiny:

  1. Always report n per arm alongside the interval. A tight CI from 500 visits per arm is usually a red flag, not a reassurance — check your variance estimator.
  2. Show the interval graphically, not just in text — a horizontal bar with the point estimate and whiskers reads faster than a sentence with two numbers in brackets.
  3. Flag novelty effects explicitly. A hero or form test (like Snake River Farms' email capture) that shows a strong week-one lift can decay as repeat visitors become "blind" to the new element — if you have the data, report day 1-7 CI separately from day 8-14 CI, since secondary engagement metrics like scroll depth and session duration often diverge from conversion rate over time in exactly this pattern.

Common questions

What sample size do I need for a reliable confidence interval on conversion rate?

For a 3% baseline CR and a target of detecting a 10% relative lift at 80% power, you typically need 15,000–17,000 visitors per arm. Lower baseline rates, smaller target effects, or higher power requirements (90%+) all push this number up substantially — often by 1.5-2x.

Can a test be statistically significant but still have an unreliable confidence interval?

Yes. A p-value under 0.05 only tells you the result is unlikely under the null — it says nothing about precision. A test can clear significance with a CI of [+0.5%, +25%] relative lift, which is technically a "win" but too imprecise to forecast revenue impact confidently.

How wide should a confidence interval be before I distrust a test result?

There's no universal cutoff, but if your CI's lower bound is within 1-2 percentage points of zero (for relative lift) or the interval spans more than 3x the point estimate, treat it as inconclusive. Re-run with a larger sample or extend test duration rather than shipping off a borderline interval.

Do Bayesian methods eliminate the risk of false positives from peeking at results early?

No, but they reduce it when paired with a pre-registered decision rule (like a fixed expected-loss threshold). Peeking without any stopping rule — Bayesian or frequentist — still inflates false positive rates; the fix is process discipline, not the choice of statistical framework alone.

Should I report confidence intervals for secondary metrics too, not just conversion rate?

Yes, especially for metrics like scroll depth, session duration, and retention rate that often move alongside conversion rate. Reporting CIs only on your primary metric while treating secondary metrics as anecdotal creates a false sense of precision about the overall experiment story.

Takeaway for your next sprint

Before you report your next test result, add one column to your results doc: the 95% CI on relative lift, calculated with the ratio method above, not just the point estimate and p-value. If the interval's lower bound is near zero or negative, don't call it a win yet — extend the test, segment by traffic source, or flag it as "directionally positive, needs replication" instead. That one habit is the difference between a roadmap built on noise and one built on effects you can actually forecast.

Read next

More from the ABWatcher blog

From the glossary

Vocabulary the article uses. See all terms →

  • Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
  • Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
  • Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
  • Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
  • Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
  • Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

See more like this

ABWatcher catches A/B tests like this across 1,000+ brands.

Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.