CRO Glossary
Experiment Hypothesis Backlog
A prioritized, running list of untested hypotheses a team maintains to feed future experiments.
An experiment hypothesis backlog is a living repository of ideas — generated from analytics, user research, support tickets, heuristic audits, or competitor analysis — that have been framed as testable hypotheses but haven't yet been built or run. It's distinct from a general experiment backlog of in-flight or scheduled tests; this is the raw idea pool that feeds it.
Teams maintain this backlog because good experiment ideas are a scarcer resource than testing capacity. Without a structured place to capture and score hypotheses, teams tend to test whatever the loudest stakeholder suggested most recently, rather than the idea most likely to move the needle. A well-kept backlog also prevents the same idea from being re-proposed and re-debated every quarter.
Entries typically include the hypothesis itself (if we do X, then Y will happen, because Z), supporting evidence, the metric it targets, and a rough prioritization score such as ICE or PIE. The backlog is reviewed regularly — often in the same meeting where past results are shared — so that learnings from closed experiments can generate new hypotheses and keep the pipeline fed.
For example, a backlog entry might read: 'If we remove the second address field on checkout, completion rate will increase, because session recordings show 18% of users abandon exactly at that field.' This sits alongside a dozen other candidate hypotheses, ranked by expected impact and effort, waiting for a slot in the experiment roadmap.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment hypothesis backlog.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.