CRO Glossary
Test Duration
The length of time an experiment runs, chosen to reach adequate sample size while covering natural cycles in user behavior.
Test duration isn't just about hitting a target sample size calculated from your minimum detectable effect and statistical power; it also needs to span enough business cycles to be representative. Running a test for only three days might catch a weekday-only pattern and miss how weekend shoppers behave completely differently, leading to a result that doesn't generalize. A common rule of thumb is to run tests for at least one to two full weeks, in increments of full weeks, so that every day of the week is equally represented in both variants. This also helps average out one-off events like a marketing email blast or a site outage that could otherwise skew a single day's numbers.
Too-short tests risk being fooled by noise or the novelty effect; too-long tests waste opportunity cost and increase the risk of external factors (seasonality, competitor actions, algorithm changes) contaminating results. Teams typically pre-register an expected duration before launch based on power calculations, and resist the urge to stop early just because the peeking problem makes an early result look exciting.
Related terms
Above vs Below Median Split
A segmentation technique that divides users into two groups at the median of a metric to compare high and low engagers.
Qualitative Feedback Loop
A recurring process of collecting and acting on user comments, session recordings, and survey responses to inform test hypotheses.
Experiment Metadata
The structured record of who, what, when, and how for a test — hypothesis, owner, dates, variants, and metrics used to track and audit it.
Pre-Test Analysis
The upfront work of validating traffic, metrics, and baseline data quality before launching an experiment.
Above-the-Fold vs Below-the-Fold Messaging Hierarchy
How the order and prominence of value-proposition messaging is split between the visible screen and the rest of the page.
Experiment Contamination
When a control or variant group is unintentionally exposed to the other condition, corrupting the comparison.
See this in the wild
ABWatcher watches how top teams apply test duration.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.