CRO Glossary
Experiment Documentation
The written record of an experiment's hypothesis, design, results, and learnings kept for future reference.
Experiment documentation is the paper trail that turns a single test into organizational knowledge. It typically captures the hypothesis, the metrics tracked, the variants shown, the audience and duration, the statistical results, and — critically — the interpretation and decision that followed. Without it, teams tend to re-run tests they've already done, forget why a past change was rolled back, or lose the nuance behind a 'winning' variant.
Good documentation is written for someone who wasn't in the room. It should explain not just what happened but why the test was run in the first place and what business question it was meant to answer. This matters most when a team scales past a handful of experimenters, or when there's turnover — six months later, nobody remembers that the checkout redesign test was contaminated by a concurrent pricing change unless it's written down.
A practical example: a growth team keeps a shared doc template with fields for hypothesis, primary and guardrail metrics, screenshot of each variant, sample size and duration, final p-value or credible interval, and a plain-English takeaway like 'Removing the trust badge hurt mobile conversion by 4%, keep as-is.' New hires can search this archive before proposing a new test, avoiding duplicated effort and surfacing tribal knowledge that would otherwise live only in someone's memory.
Documentation also protects against revisionist history — the temptation to reinterpret a flat or negative result as a win once it's inconvenient. A dated, timestamped record makes it much harder to quietly change the story after the fact.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment documentation.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.