CRO Glossary
Experiment Prioritization Framework
A structured method (like PIE or ICE) for scoring and ranking test ideas so teams run the highest-value experiments first.
When a team has more test ideas than time to run them, an experiment prioritization framework gives a repeatable, less-political way to decide what goes first. The most common models score each idea across a few dimensions — for example ICE (Impact, Confidence, Ease) or PIE (Potential, Importance, Ease) — and combine them into a single number that ranks the backlog. This turns "whoever shouts loudest gets their test built" into a documented, defensible process.
The value isn't really the precision of the score itself — these are rough, subjective estimates — but the discipline of forcing a team to articulate why an idea might work, how confident they are, and what it costs to build. That conversation alone often kills weak ideas before they consume engineering time.
For example, a growth team might score "redesign the pricing page" as high impact, medium confidence, high effort, versus "add a trust badge near checkout" as medium impact, high confidence, very low effort. Even though the pricing redesign might move more revenue, the trust badge could win the prioritization score because it's fast to ship and test, letting the team learn quickly and reinvest that time into bigger bets.
Frameworks should be revisited periodically since scoring criteria (like what counts as "high confidence") drift as a team matures and accumulates more historical test data to calibrate against.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment prioritization framework.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.