Estimand

An estimand is the formal definition of what exactly you're trying to measure — the target parameter — as distinct from the estimator (the statistical method used to compute it) and the estimate (the actual number you get). In experimentation, sloppy thinking about the estimand is a quiet source of misleading results: two teams can both report 'the effect of the new checkout flow' while actually measuring different things, such as the effect on all users randomized versus the effect only on users who actually saw the new flow.

Getting the estimand right matters because it determines your analysis population, your metric window, and how you handle dilution from non-compliance (users assigned to treatment who never actually experienced it). A vague estimand leads to p-hacking-adjacent decisions after the fact — picking whichever population or window makes the result look best — even without any bad intent.

Concretely, imagine testing a new onboarding email for new signups. One valid estimand is the intent-to-treat effect: the average effect of being assigned to the new email, across everyone randomized, including people whose email bounced or went to spam. Another is the effect on the compliers: users who were assigned and actually opened the email. These can produce very different numbers — the first is diluted by non-delivery, the second is cleaner but requires assumptions to estimate without bias (e.g., instrumental variables).

Best practice is to write the estimand explicitly in the experiment design document before launch: whose behavior, over what window, compared to what counterfactual, measured how. It sounds academic, but it's the difference between a result that replicates and one that quietly means something different every time someone re-pulls the numbers.