CRO Glossary
Experiment Velocity
The rate at which a team designs, launches, and concludes experiments over a given period.
Experiment velocity is a program-level metric — often expressed as tests shipped per month or per quarter — used to measure how efficiently an organization is learning through experimentation. High velocity generally correlates with faster compounding gains, but only if quality is held constant; a team that rushes tests without adequate test duration or statistical power can post an impressive velocity number while generating unreliable, non-repeatable results.
Velocity is most useful as a leading indicator of program health and organizational buy-in (are there enough engineers, enough traffic, enough decision-makers signed off to keep tests moving), rather than as a proxy for impact. Many mature CRO teams track velocity alongside win rate and average lift per test, since a program can have high velocity and low win rate, or vice versa, and each combination suggests a different intervention — more ideation, better prioritization, or more rigorous QA.
Example: a team increases experiment velocity from 2 tests per month to 8 by building a self-serve experimentation platform, but discovers their win rate drops because many of the new tests are underpowered. They then set a minimum sample size policy to keep velocity high without sacrificing reliability.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment velocity.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.