Experiment Design Document

An experiment design document (sometimes called a test brief or experiment brief) is the artifact a team writes before running a test, laying out the hypothesis, the primary and guardrail metrics, the target audience or segment, the randomization unit, expected sample size, planned duration, and how the results will be analyzed. It exists to force clarity upfront, rather than deciding what counts as success after peeking at the data.

Practitioners care about this because it prevents two common failure modes: shifting the goalposts after seeing partial results, and running tests that nobody can act on because the question was never precisely defined. A good design document also serves as institutional memory — six months later, someone can read it and understand exactly what was tested, why, and what was supposed to happen.

For example, before testing a new checkout flow, a team would document the hypothesis ('removing the coupon code field will reduce abandonment because it currently invites cart comparison-shopping'), the primary metric (checkout completion rate), guardrails (average order value, refund rate), the minimum detectable effect, and the planned sample size and runtime. If the test later shows a surprising result, the document lets the team check whether the original plan was sound rather than rationalizing after the fact.

Many organizations require sign-off on this document before a test is allowed to launch, precisely because it discourages post-hoc metric shopping and p-hacking down the line.