CRO Glossary
Estimand
The precise quantity a test is designed to estimate, defined before the experiment runs, such as average treatment effect on exposed users.
An estimand is the formal definition of what exactly you're trying to measure — the target parameter — as distinct from the estimator (the statistical method used to compute it) and the estimate (the actual number you get). In experimentation, sloppy thinking about the estimand is a quiet source of misleading results: two teams can both report 'the effect of the new checkout flow' while actually measuring different things, such as the effect on all users randomized versus the effect only on users who actually saw the new flow.
Getting the estimand right matters because it determines your analysis population, your metric window, and how you handle dilution from non-compliance (users assigned to treatment who never actually experienced it). A vague estimand leads to p-hacking-adjacent decisions after the fact — picking whichever population or window makes the result look best — even without any bad intent.
Concretely, imagine testing a new onboarding email for new signups. One valid estimand is the intent-to-treat effect: the average effect of being assigned to the new email, across everyone randomized, including people whose email bounced or went to spam. Another is the effect on the compliers: users who were assigned and actually opened the email. These can produce very different numbers — the first is diluted by non-delivery, the second is cleaner but requires assumptions to estimate without bias (e.g., instrumental variables).
Best practice is to write the estimand explicitly in the experiment design document before launch: whose behavior, over what window, compared to what counterfactual, measured how. It sounds academic, but it's the difference between a result that replicates and one that quietly means something different every time someone re-pulls the numbers.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply estimand.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.