CRO Glossary
Experiment Prioritization
The process of ranking test ideas by expected impact, confidence, and effort before allocating experimentation resources.
Experiment prioritization is how teams decide which of many possible tests to run first, given limited traffic, engineering time, and design bandwidth. Because most organizations have far more hypotheses than they can test in a given quarter, some structured way of ranking ideas prevents the roadmap from being driven purely by whoever argues loudest in a meeting.
Practitioners typically score each idea against a few dimensions: how much it could move the target metric, how confident the team is in the underlying hypothesis, and how costly it is to build and QA. Frameworks like ICE or PIE are common shorthand for this, but the specific scoring model matters less than having a consistent, revisitable process that the whole team trusts.
For example, a growth team might have twenty ideas for the checkout flow but capacity to ship only three tests a month. Prioritization forces an explicit tradeoff: a high-effort redesign of the payment step might be deprioritized in favor of two cheaper copy and layout tweaks that are faster to validate and free up learnings for the bigger bet later.
Done well, prioritization also surfaces dependencies and sequencing issues, such as two ideas that would contaminate the same page if run simultaneously, before they turn into wasted engineering effort.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment prioritization.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.