Experiment Backlog

An experiment backlog is the queue of test ideas a team has identified but not yet launched, typically ranked using some combination of expected impact, confidence in the hypothesis, and effort to build. It functions like a product backlog, but instead of features it holds hypotheses: 'simplifying the checkout form will reduce drop-off,' 'adding social proof near the CTA will increase sign-ups,' and so on.

Maintaining a backlog matters because experimentation teams generate ideas faster than they can test them, especially once qualitative research, heatmaps, and stakeholder requests start feeding in. Without a structured backlog, prioritization tends to default to whoever shouts loudest or whichever executive raised an idea last, rather than the ideas with the best expected return. A well-run backlog forces explicit trade-offs and keeps the roadmap defensible.

Many teams score backlog items using frameworks like PIE (Potential, Importance, Ease) or ICE (Impact, Confidence, Ease), assigning rough numeric scores to each dimension and sorting by the total. For example, a hypothesis with high potential impact but low confidence and high build effort might rank below a smaller but cheap, high-confidence test that can ship this week.

A healthy backlog is reviewed regularly, since ideas can become stale as the product changes, and completed or invalidated tests should be removed or archived along with what was learned, so the same idea doesn't get proposed and re-tested from scratch a year later.