Sequential Testing vs. Fixed-Horizon A/B Tests: When to Use Each

I've shipped tests that we killed on day two because the early numbers looked so bad it felt irresponsible to keep serving the variant. I've also shipped tests that looked like clear winners on day four, that we let ride to day fourteen anyway, and watched the "winner" evaporate back to a dead heat. Both of those instincts were right in one context and dead wrong in the other. The difference wasn't the product, the traffic, or the team — it was which stopping rule we were actually using, whether we admitted it or not.

Most teams pick a testing methodology by default, usually whatever their tool ships with, and never revisit it. That's a mistake. Sequential and fixed-horizon testing have different math, different failure modes, and different ideal use cases. Getting this wrong either burns weeks of traffic you didn't need to burn, or leads you to ship changes that don't actually hold up.

The core mechanical difference

Fixed-horizon testing is what most people mean when they say "A/B test." You calculate a required sample size up front based on your baseline conversion rate, minimum detectable effect (MDE), and desired power — then you don't look at the result, or at least don't act on it, until you hit that number. The statistics are clean because you're doing exactly one test of significance at exactly one point in time.

Sequential testing flips that. You define a stopping boundary — using something like an always-valid p-value, a group sequential design, or a Bayesian probability threshold — and you're allowed to check the data continuously (or in scheduled batches) and stop as soon as the boundary is crossed, in either direction. Amplitude's comparison frames this well: sequential methods are built specifically to let you peek without inflating your false positive rate, which a naive fixed-horizon test absolutely does not tolerate.

That last point is the one engineers get wrong constantly. If you're running a fixed-horizon test and checking the dashboard every day, eyeballing the p-value, and stopping the moment it crosses 0.05 — you're not running a fixed-horizon test anymore. You're running an ad hoc sequential test with no correction, and your actual false positive rate is nowhere near 5%. It can be 20-30% depending on how often you peek. This is, in my experience, the single most common statistical error on growth teams — not picking the wrong method, but silently switching methods mid-flight without adjusting the math.

Sample size and time-to-signal tradeoffs

Fixed-horizon tests need you to commit to a sample size before you start, which forces discipline but also forces waiting. If your MDE is small — say you're trying to detect a 3% lift in signup conversion — you need a large, precise sample, and there's no way to shortcut that with a fixed design. You commit to the full run or you don't run it.

Sequential tests shine on the other end of that spectrum. Amplitude points out that sequential methods are especially advantageous when you're testing large, structural changes and your business has a smaller customer base — situations where the gap between your minimum detectable effect and maximum plausible effect is wide. If a redesign is going to move the needle at all, it's probably going to move it a lot, and sequential testing lets you detect that big swing early and stop, instead of grinding through a sample size calculated for detecting a much smaller effect.

The practical numbers: on a typical SaaS trial-to-paid funnel, a fixed-horizon test targeting an 8% relative lift at 80% power might need two to three weeks of full traffic. A well-tuned sequential design testing the same funnel for the same effect size can often call a clear win or a clear loss in 30-40% less time — not because the math is more powerful, but because it doesn't force you to wait for a fixed endpoint when the evidence is already overwhelming at day nine.

Risk profiles: false positives vs. opportunity cost

Every stopping rule is really a decision about which kind of risk you're willing to eat.

Fixed-horizon testing protects you from false positives at the cost of opportunity. You will not ship a fluke winner, because you refused to look until the sample was complete. But you also won't stop a clear loser early — you'll keep serving a variant that's actively hurting conversion for the full duration, because stopping early breaks the statistical guarantee.

Sequential testing protects you from wasted traffic and slow bad-news at the cost of needing tighter engineering discipline. Done properly with a real sequential method (group sequential design, mixture sequential probability ratio tests, or always-valid inference), your false positive rate is still controlled — that's the entire point of the math. Done improperly, by just checking a standard t-test repeatedly, you're accumulating exactly the risk fixed-horizon testing was designed to prevent.

Kameleoon's writeup describes a pattern I've used at more than one company: run sequential tests to screen ideas fast and cheaply, then re-run the promising ones through a fixed-horizon design before you commit engineering resources to a full rollout. The sequential pass is your triage. The fixed-horizon pass is your confirmation. Treating either one as sufficient on its own is where teams get burned — sequential-only shops ship too many things that don't replicate, and fixed-horizon-only shops move too slowly to catch enough ideas in the first place.

There's a third failure mode worth naming: novelty and seasonality effects. A change that pops in the first three days of a sequential test can be entirely a novelty spike from your existing users noticing something new, not a durable behavioral shift. This is precisely why the confirmation pass matters — you're not just re-checking significance, you're checking that the effect survives contact with a full week-over-week cycle, including your slow Tuesday and your promo-driven Saturday.

Infrastructure requirements: what "always-on" actually costs

Sequential testing is often pitched as the more modern, more sophisticated option, and infrastructure vendors love to sell it that way. It's worth being honest about what it actually requires operationally, not just statistically.

You need an experimentation platform that computes valid sequential boundaries in real time, not a dashboard that just shows a rolling p-value (which, again, is not the same thing and does not control error rates). Spotify's experimentation platform is a good real-world reference point here — they adopted group sequential tests specifically because their data infrastructure processes results in batches, and GSTs are designed to work cleanly with periodic, rather than truly continuous, data checks. That's a useful model for most teams: you don't need millisecond-level continuous monitoring, you need a small number of well-defined checkpoints (say, daily or every 48 hours) with pre-registered boundaries at each one.

Fixed-horizon testing, by contrast, needs almost none of this. A sample size calculator, a commitment to not look early, and a single significance test at the end. The infrastructure cost is nearly zero, which is exactly why it remains the default for teams without a dedicated experimentation platform or data science support.

If your team is choosing between building sequential testing infrastructure and just running more fixed-horizon tests in parallel, run the math on your actual test volume first. Below roughly 10-15 concurrent experiments a quarter, the operational overhead of doing sequential testing correctly usually isn't worth it. Above that, especially if you're testing across multiple funnels or segments simultaneously, the time savings compound fast enough to justify the build.

What this looks like across live experiments

Across the tests we track at high-converting brands, a pattern shows up repeatedly: bold, structural changes — pricing page rearchitectures, checkout flow consolidations, onboarding redesigns — tend to get pulled or promoted fast, often inside a week, which is the signature of either a sequential design or a team that's comfortable calling a fixed-horizon test early because the effect is enormous. Smaller, incremental tests — button copy, badge placement, form field ordering — tend to run longer, sometimes three to four weeks, which is consistent with fixed-horizon designs chasing a small MDE. The stopping behavior alone tells you a lot about what kind of effect a team believes it's testing, before you even see the result.

What I would actually do next

Don't pick a testing philosophy — pick it per experiment. This sprint, sort your test backlog into two buckets: structural bets where you expect a large effect (or nothing), and incremental bets chasing a small, uncertain lift. Run the structural bets sequentially, with a real pre-defined stopping boundary, so you can kill losers fast and stop burning traffic on things that clearly aren't working. Run the incremental bets fixed-horizon, with a sample size calculated up front, and hold your nerve not to peek. If a sequential test on a big bet shows a strong signal within the first third of your planned duration, don't ship it immediately — queue a short fixed-horizon confirmation run before it hits your roadmap as "done." That one habit — triage fast, confirm slow — will save you more false wins than any dashboard upgrade.