CRO Glossary
Experiment Design Document
A written plan that specifies an experiment's hypothesis, metrics, audience, duration, and analysis plan before launch.
An experiment design document (sometimes called a test brief or experiment brief) is the artifact a team writes before running a test, laying out the hypothesis, the primary and guardrail metrics, the target audience or segment, the randomization unit, expected sample size, planned duration, and how the results will be analyzed. It exists to force clarity upfront, rather than deciding what counts as success after peeking at the data.
Practitioners care about this because it prevents two common failure modes: shifting the goalposts after seeing partial results, and running tests that nobody can act on because the question was never precisely defined. A good design document also serves as institutional memory — six months later, someone can read it and understand exactly what was tested, why, and what was supposed to happen.
For example, before testing a new checkout flow, a team would document the hypothesis ('removing the coupon code field will reduce abandonment because it currently invites cart comparison-shopping'), the primary metric (checkout completion rate), guardrails (average order value, refund rate), the minimum detectable effect, and the planned sample size and runtime. If the test later shows a surprising result, the document lets the team check whether the original plan was sound rather than rationalizing after the fact.
Many organizations require sign-off on this document before a test is allowed to launch, precisely because it discourages post-hoc metric shopping and p-hacking down the line.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment design document.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.