CRO Glossary
Experiment Debt
The accumulated backlog of unanalyzed, unshipped, or undocumented experiments that erodes an organization's ability to learn from testing.
Experiment debt builds up when a team runs more tests than it has capacity to properly analyze, document, and act on. Symptoms include experiments left running past their planned duration with no read-out, winning variants that never get shipped to 100%, and results that live only in someone's head or a stale spreadsheet instead of a searchable repository.
This matters because the value of experimentation compounds through institutional knowledge — knowing what's already been tried on the pricing page, or which audience segments respond to urgency messaging, saves future teams from re-running redundant tests. When experiment debt piles up, organizations lose that compounding effect and start repeating past mistakes, sometimes literally re-testing the same hypothesis a competitor team already disproved a year earlier.
A typical warning sign is a testing program with an impressive experiment count but a low percentage of experiments that actually influenced a shipped decision. Paying down experiment debt usually means investing in a lightweight tracking plan or shared results archive, and building read-out and archiving steps into the definition of "done" for any test, not just the launch step.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment debt.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.