CRO Glossary
Experiment Backlog
A prioritized list of hypotheses and planned tests waiting to be run, ranked by expected impact and effort.
An experiment backlog is the queue of test ideas a team has identified but not yet launched, typically ranked using some combination of expected impact, confidence in the hypothesis, and effort to build. It functions like a product backlog, but instead of features it holds hypotheses: 'simplifying the checkout form will reduce drop-off,' 'adding social proof near the CTA will increase sign-ups,' and so on.
Maintaining a backlog matters because experimentation teams generate ideas faster than they can test them, especially once qualitative research, heatmaps, and stakeholder requests start feeding in. Without a structured backlog, prioritization tends to default to whoever shouts loudest or whichever executive raised an idea last, rather than the ideas with the best expected return. A well-run backlog forces explicit trade-offs and keeps the roadmap defensible.
Many teams score backlog items using frameworks like PIE (Potential, Importance, Ease) or ICE (Impact, Confidence, Ease), assigning rough numeric scores to each dimension and sorting by the total. For example, a hypothesis with high potential impact but low confidence and high build effort might rank below a smaller but cheap, high-confidence test that can ship this week.
A healthy backlog is reviewed regularly, since ideas can become stale as the product changes, and completed or invalidated tests should be removed or archived along with what was learned, so the same idea doesn't get proposed and re-tested from scratch a year later.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment backlog.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.