CRO Glossary
Pre-Experiment Bucketing Audit
A review process confirming users are being randomly and evenly assigned to experiment variants before launch.
A pre-experiment bucketing audit is a quality-control step where a team verifies that the randomization and assignment logic of an experimentation platform is actually working as intended before a test goes live to real traffic. It's distinct from monitoring sample ratio mismatch after the fact — this audit happens proactively, often using synthetic traffic, QA accounts, or a dry run, to catch assignment bugs before they contaminate real data.
Teams care about this because bucketing bugs are common and expensive: a hashing function with a subtle bias, a caching layer that pins users to a stale variant, or a feature flag misconfiguration can all silently skew who sees what. Catching this after a test has run for two weeks means throwing away the data; catching it beforehand means a five-minute fix.
A typical audit involves generating thousands of simulated user IDs, running them through the assignment logic, and checking that the resulting split matches the intended allocation (e.g., 50/50) within statistical tolerance. Teams also check edge cases: logged-out vs logged-in users, users switching devices, and users who enter the funnel through different pages.
For example, before launching a checkout redesign test, a team might run a bucketing audit and discover that mobile app users are being assigned differently than mobile web users due to a platform-specific SDK bug — a problem invisible in the metrics until it's already biased the results.
Related terms
Shrinkage Estimator
A statistical adjustment that pulls noisy per-variant or per-segment estimates toward an overall average to reduce overfitting to randomness.
Flicker Test
A diagnostic check to confirm whether users briefly see the original page before a variant renders in client-side experiments.
Above-Fold Exclusion Rule
A test-design rule that deliberately prevents changes to above-the-fold elements so an experiment isolates below-the-fold effects.
Flicker-Free Testing
A technical setup that prevents users from briefly seeing the original page before a test variant loads.
Orthogonal Test Design
A test structure where multiple variables are changed independently so their individual effects can be measured without interference.
Experiment Decay
The gradual erosion of a winning test's uplift over time after it is fully rolled out.
See this in the wild
ABWatcher watches how top teams apply pre-experiment bucketing audit.
Live A/B tests at 1,000+ high-converting brands, with plain-English hypothesis and takeaway.