The ABWatcher blog

Frequentist vs. Bayesian A/B Testing: Choosing the Right Model for Landing Pages

A practical comparison of frequentist and Bayesian A/B testing for landing pages — sample size math, stopping rules, and how to pick the right framework for your traffic.

Maya Patel

Senior CRO Strategist · · Updated

Most CRO teams inherit their statistical framework from whatever tool they bought, not from a deliberate choice. That's a problem, because the choice between frequentist and Bayesian testing changes how you calculate sample size, when you're allowed to peek at results, and what "significant" even means. Get it wrong and you'll either ship losers with false confidence or sit on winners for three extra weeks waiting for a p-value that was never going to move.

TL;DR

  • Frequentist testing requires a fixed sample size calculated in advance — peeking early inflates false-positive rates above the nominal 5%.
  • Bayesian testing supports continuous monitoring and early stopping, reporting results like "92% probability B beats A."
  • At an MDE of 5% or higher, several experienced practitioners default to Bayesian for faster directional reads (Convert.com).
  • Frequentist remains the standard in regulated or audit-heavy environments because "significant" has a fixed, defensible definition (Kameleoon).
  • ABWatcher detected live pricing, hero, and layout tests at SignNow, Disney+, Lyft, Prismatic, and Dropbox Sign between 8 Jul and 8 Aug 2026 — all plausible candidates for either framework depending on traffic and stakes.
  • Low-traffic landing pages (under ~5,000 weekly visitors) generally favor Bayesian methods to avoid indefinitely inconclusive frequentist tests.

What's the core philosophical difference between the two approaches?

Frequentist testing asks "how surprising is this data if there's no real effect?" while Bayesian testing asks "what's the probability the effect is real, given everything we know?" The frequentist approach treats the true conversion rate as a fixed but unknown constant and asks how likely your observed data (or something more extreme) would be under a null hypothesis of no difference — that's the p-value. It never tells you the probability that your variant is better; it tells you the probability of your data assuming it isn't.

Bayesian testing flips this. It starts with a prior belief about likely conversion rates, updates that belief as data arrives, and outputs a posterior probability — something you can state plainly, like "there's an 87% chance Variant B outperforms Control." That's closer to what stakeholders actually want to hear, and it's a big reason Bayesian methods are gaining ground in CRO teams that need to communicate results to non-statisticians on tight sprint cycles.

The tradeoff is auditability versus intuitiveness. Frequentist p-values are the accepted standard in medical research and regulated industries, and "statistically significant" has a fixed, widely understood threshold. Bayesian posteriors depend on your choice of prior, which introduces a subjective element that skeptical stakeholders — or compliance teams — may push back on.

How does sample size calculation actually change between the two?

Frequentist tests require you to fix your sample size and test duration before launch, based on baseline conversion rate, minimum detectable effect (MDE), and desired power — Bayesian tests let you monitor continuously and stop when the posterior crosses a confidence threshold you set in advance. This isn't a minor implementation detail; it changes your entire test-planning workflow.

For a frequentist test, you'd typically use a baseline rate, an MDE, 80% power, and a 95% confidence level to calculate required sample size before launch — and you don't touch the results until you hit that number. A worked example from Mastercard's Dynamic Yield guidelines illustrates why this matters: a landing page converting at 1% with $20 average order value might see a variant that lifts conversion to 1.2% but pulls in fewer high-value purchasers, dropping average order value to $19. Revenue per visitor can move in a direction opposite to what your headline conversion metric suggests — which is exactly the kind of secondary effect that a rigid, single-metric frequentist test can miss if you're not tracking revenue per visitor alongside conversion rate.

Bayesian tests don't eliminate the sample size question, but they handle it more flexibly. You define your prior, set a decision threshold (e.g., stop when there's a 95% probability B beats A, or when expected loss from choosing wrong drops below an acceptable threshold), and check in as often as you like without inflating your false-positive rate the way repeated frequentist peeking does. This is particularly valuable for teams with variable weekly traffic who can't commit to a fixed test window three weeks out.

Dimension Frequentist Bayesian
Core question P(data | no effect) P(effect | data)
Sample size Fixed in advance Flexible, can update in-flight
Peeking at results Inflates false positives Statistically valid
Output Significant / not significant Probability variant wins (e.g., 92%)
Best traffic fit High-traffic, well-planned tests Lower or variable traffic
Stakeholder communication Requires stats literacy Intuitive, plain-language
Audit trail Strong, industry standard Depends on documented priors

When does frequentist testing still win?

Frequentist testing wins when you need a defensible, audit-ready result and have enough traffic to hit your pre-calculated sample size within a reasonable window. As founder Lucia van den Brink put it, frequentist framing is "easy to explain if a test is either a win, loss (learning) or non-significant," and it remains the standard across medical research and many regulated fintech contexts (Kameleoon). If your organization needs to defend a launch decision to legal, compliance, or a skeptical VP with a stats background, "p < 0.05 at n=48,000" is harder to argue with than a posterior probability that depends on a prior someone chose.

Frequentist also suits large-scale, high-traffic pricing and pages where you can hit adequate sample size in days, not months. Disney+'s pricing-page test — detected 20 Jul 2026, splitting the "Choose your plan" section between a card layout with visible SELECT buttons and a text/checkmark comparison table — is the kind of test Disney+ can run at frequentist scale given its traffic volume, allowing a fixed-duration test to reach significance quickly without needing continuous monitoring shortcuts.

When should teams reach for Bayesian instead?

Bayesian testing suits lower-traffic landing pages, continuous monitoring workflows, and situations where an MDE of 5% or higher makes a fast directional read more valuable than a slow, airtight one. One CRO practitioner interviewed by Convert.com put it directly: "When the MDEs are 5% or higher, I prefer to go Bayesian. It gives directional answers like 'there's a 92% chance the variant is better.'" That's a meaningful signal for teams that need to ship decisions on a two-week sprint cadence rather than wait out a multi-week fixed-horizon frequentist test.

Copy and headline tests on mid-traffic pages are a natural fit here. Lyft's subheadline test on its "Choose how to ride" section — detected 8 Jul 2026, alternating between "Find a ride that suits your mood and budget" and "Hit the road, or the rails. Find a ride that fits your style and budget" within the same Chrome engine — is a low-structural-risk copy variant where a Bayesian read on early signal could support a faster ship/kill decision than waiting for frequentist significance, particularly if Lyft is testing across multiple concurrent copy variants and needs to reallocate traffic quickly.

Similarly, Prismatic's free-trial subheadline test — detected 5 Aug 2026, splitting "14-day free trial" against generic "your free trial" phrasing across multiple browser engines — represents exactly the kind of single-variable copy test where continuous Bayesian monitoring lets Prismatic stop as soon as the posterior probability clears a working threshold, rather than committing to a rigid sample size that may take weeks to accumulate on a B2B trial-signup flow with lower absolute traffic than a consumer app.

How do multi-element structural tests complicate the choice?

Structural and layout tests with multiple interacting elements often need larger samples and more conservative frameworks than single-variable copy tests, regardless of which statistical approach you pick. SignNow's hero test — detected 8 Aug 2026, splitting between a single-CTA "Get your first document signed" hero and a two-column hero with an added client-logo strip (Walmart, ExxonMobil, Apple, Comcast, Facebook, FedEx) and a "Why choose airSlate SignNow" section — bundles multiple changes into one variant, which makes attribution harder under either framework. When SignNow tests a compound variant like this, isolating which element drove the lift (logo strip vs. CTA copy vs. added trust section) benefits from either a larger frequentist sample with proper power for the interaction effect, or a Bayesian model that explicitly accounts for multiple correlated metrics.

Dropbox Sign's pricing-page test — detected 5 Aug 2026, where some Chrome visits show an "Enhance your plan with add-ons" section (Replay, Extended Version History, Sign, DocSend, Reclaim.ai) between the plan cards and FAQ, while others omit it entirely — is a similar structural test where the decision metric matters as much as the framework. If Dropbox Sign is measuring plan selection rate alongside average revenue per visitor (add-on attach rate could offset a small drop in base-plan conversion), a Bayesian approach that models both metrics jointly may surface the tradeoff faster than two separate frequentist tests run in sequence.

Which framework fits your team this sprint?

Run an ICE-style filter before you pick a framework: score each pending test on Impact (revenue or conversion delta at stake), Confidence (how much you already believe in the change), and Ease (how fast you can reach adequate sample size). High-traffic, high-stakes pricing tests with compliance exposure — like a fixed-plan pricing page — lean frequentist. Lower-traffic copy and hero tests where speed matters more than airtight proof — like subheadline or CTA wording — lean Bayesian. If you're running both types concurrently, as the Mida comparison notes, frequentist is simpler to audit while Bayesian works better for continuous monitoring and early stopping — so a hybrid stack by test type, rather than a single company-wide standard, is often the most defensible answer.

Common questions

Is Bayesian A/B testing less rigorous than frequentist testing?

No — it's a different rigor, not less rigor. Bayesian testing requires the same discipline in defining priors, decision thresholds, and expected-loss tolerances upfront that frequentist testing requires in defining sample size and power. The risk is choosing an unjustified prior, which is why documenting prior assumptions before launch matters as much as pre-registering a frequentist sample size.

Can I switch from frequentist to Bayesian mid-test?

Not cleanly. Switching frameworks mid-flight invalidates the guarantees either method offers — frequentist error rates assume no peeking, and Bayesian posteriors assume continuous, honest updating from the start. Decide your framework during test design, not after you've already seen early results trending a direction you like.

What sample size do I need for a Bayesian test versus frequentist?

Bayesian tests don't require a fixed pre-calculated sample size, but they still perform better with more data — informative priors reduce the data needed to reach a confident posterior. Frequentist tests require a specific number calculated from baseline rate, MDE, and power before you can call a result valid, typically in the thousands of visitors per variant for a 5% MDE.

Do frequentist and Bayesian methods ever agree?

Yes, frequently — with large sample sizes and weakly informative priors, Bayesian posteriors and frequentist confidence intervals often converge to similar conclusions. Disagreement is most likely in small-sample, low-traffic, or early-stopping scenarios, which is exactly where the choice of framework matters most.

Which industries default to frequentist testing?

Regulated sectors — fintech, healthcare, insurance — tend to default to frequentist testing because "statistically significant" has a fixed, externally recognized definition that holds up to compliance review. Consumer and SaaS teams optimizing for shipping velocity more often adopt Bayesian methods for lower-stakes copy and layout tests.

Before your next test kicks off, write down which framework you're using and why — tied to traffic volume, stakes, and how fast you need to decide — rather than defaulting to whatever your testing tool ships with by default. That one documentation step prevents the most common failure mode: switching interpretive lenses mid-test because the early data wasn't going your way.

Read next

More from the ABWatcher blog

From the glossary

Vocabulary the article uses. See all terms →

  • Shrinkage Estimator— A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
  • Flicker Test— A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
  • Pre-Experiment Bucketing Audit— A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
  • Above-Fold Exclusion Rule— A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
  • Flicker-Free Testing— A technical setup that prevents users from briefly seeing the original page before a test variant loads.
  • Orthogonal Test Design— A test structure where multiple variables are changed independently so their individual effects can be measured without interference.

See more like this

ABWatcher catches A/B tests like this across 1,000+ brands.

Watch live experiments at 1,000+ high-converting brands, complete with hypothesis and takeaway. 7 days of Pro, free, no card, then Free forever for ten watched companies.