CRO Glossary
Sequential Testing
A statistical method that allows valid checking of test results continuously, without inflating false-positive rates.
Sequential testing methods (such as group sequential designs or always-valid confidence sequences) are built specifically to let you look at results as they come in and stop early if there's a clear winner or loser, without invalidating your statistics. This directly solves the peeking problem: with a standard fixed-sample significance test, checking results daily and stopping the moment p < 0.05 dramatically increases the real false-positive rate, sometimes to 20-30% instead of the intended 5%.
Modern experimentation platforms increasingly use sequential methods by default so that product teams can monitor dashboards in real time and make faster ship/kill decisions without a statistician manually correcting for repeated looks. The tradeoff is usually a slightly wider confidence interval or a need for a bit more data compared to a single, pre-planned fixed-horizon test, in exchange for the freedom to peek safely.
For example, a sequential test might let a team stop a checkout redesign test after 4 days instead of the planned 14 because the sequential boundary was crossed decisively, while a naive fixed-horizon test would have required waiting the full 14 days regardless of how obvious the result looked early on.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply sequential testing.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.