CRO Glossary
Pre/Post Analysis
A method of estimating a change's impact by comparing metrics before and after launch, without a control group.
Pre/post analysis compares a metric in the period before a change shipped to the period after, and attributes the difference to the change. It's the fastest way to get a read on impact when you can't or didn't run a controlled experiment, but it's also the weakest form of evidence because it conflates the treatment effect with everything else that changed over time: seasonality, marketing spend, macro trends, competitor moves, or even the day of the week.
Teams reach for pre/post analysis when a feature was rolled out to 100% of users immediately, when an A/B test wasn't technically feasible, or when stakeholders want a quick sanity check before a more rigorous read-out arrives. It's genuinely useful as a triangulation signal alongside a proper experiment, but dangerous as a sole source of truth because any coincidental trend gets credited to the launch.
For example, a team redesigns checkout and sees conversion rate jump 8% the week after launch. Without a holdout group, they can't tell how much of that lift came from the redesign versus a simultaneous drop in payment gateway errors or a seasonal spike in demand. Analysts often try to patch this with difference-in-differences techniques, comparing the pre/post delta against a similar unaffected segment or market, which at least partially controls for shared external trends.
The core risk practitioners should flag: pre/post comparisons are especially vulnerable to regression to the mean and to any external event that happens to coincide with launch, so conclusions should be stated with much lower confidence than a randomized test.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
See this in the wild
ABWatcher watches how top teams apply pre/post analysis.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.