CRO Glossary
Experiment Scorecard
A standardized summary document comparing an experiment's results against its primary, secondary, and guardrail metrics to drive a ship/no-ship decision.
An experiment scorecard is the standardized artifact used at the end of a test to decide whether to ship, iterate, or kill a variant. It typically lists the primary success metric with its result and confidence interval, secondary metrics for context, guardrail metrics to confirm nothing broke, and a final recommendation with reasoning. Unlike a free-form readout, a scorecard uses a consistent template across all experiments so results are comparable and decisions are auditable later.
The value of a scorecard is mostly organizational: it prevents cherry-picking a favorable secondary metric when the primary metric is flat or negative, and it creates a paper trail so that six months later someone can see not just what shipped but why. It also forces discipline — if the template requires stating the pre-registered primary metric and the actual observed result side by side, it's harder to quietly shift the goalposts after seeing the data.
A typical scorecard for a checkout redesign test might show: primary metric (purchase conversion rate) up 3.2% with a confidence interval that excludes zero; secondary metric (average order value) flat; guardrail metric (page load time) unchanged; recommendation: ship. If the guardrail had regressed, the scorecard would flag that even a conversion lift shouldn't automatically mean shipping.
Many experimentation platforms auto-generate a scorecard from the underlying data, but the discipline of defining which metrics belong in which section happens at experiment design time, not after the results come in.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply experiment scorecard.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.