CRO Glossary
Sequential Testing
A statistical method that allows valid checking of test results continuously, without inflating false-positive rates.
Sequential testing methods (such as group sequential designs or always-valid confidence sequences) are built specifically to let you look at results as they come in and stop early if there's a clear winner or loser, without invalidating your statistics. This directly solves the peeking problem: with a standard fixed-sample significance test, checking results daily and stopping the moment p < 0.05 dramatically increases the real false-positive rate, sometimes to 20-30% instead of the intended 5%.
Modern experimentation platforms increasingly use sequential methods by default so that product teams can monitor dashboards in real time and make faster ship/kill decisions without a statistician manually correcting for repeated looks. The tradeoff is usually a slightly wider confidence interval or a need for a bit more data compared to a single, pre-planned fixed-horizon test, in exchange for the freedom to peek safely.
For example, a sequential test might let a team stop a checkout redesign test after 4 days instead of the planned 14 because the sequential boundary was crossed decisively, while a naive fixed-horizon test would have required waiting the full 14 days regardless of how obvious the result looked early on.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply sequential testing.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.