Personalization

Personalization goes a step beyond a single-variant A/B test by serving different experiences to different user segments based on characteristics like geography, device, referral source, past purchase behavior, or account plan tier. Rather than asking 'which single version wins for everyone,' personalization asks 'which version wins for this type of user.'

Experimentation programs often discover personalization opportunities through segment analysis: a test might show a flat or negative result overall, but a strongly positive result within a specific segment, like mobile users or first-time visitors, that gets masked when looking only at the aggregate. That's a signal the flat result might really be two divergent stories, similar in spirit to Simpson's paradox, where an aggregate trend hides opposite patterns in subgroups.

A common example is showing returning customers a 'welcome back' message with their previously viewed products, while first-time visitors see a broader value proposition and trust signals. Because personalization multiplies the number of experiences being served and evaluated, teams need larger sample sizes per segment and clear rules for how segments are defined ahead of time, to avoid p-hacking through after-the-fact segment slicing.