Cohort Analysis

Cohort analysis groups users based on when they first did something — signed up, made a first purchase, entered a test — and then tracks how that specific group behaves over following days, weeks, or months. This is different from looking at a single blended metric across all users at once, which can hide important trends, similar to how Simpson's paradox can hide a reversal of a trend within aggregate data.

In experimentation, cohort analysis is especially useful for judging whether a treatment's effect holds up over time or fades, which helps distinguish a genuine improvement from a novelty effect. Comparing the week-one cohort's retention to the week-four cohort's retention, both measured a consistent number of days after their first exposure, is a much fairer comparison than looking at a single snapshot date where cohorts are of very different ages.

For example, if a new onboarding flow is tested, looking only at 'active users today' might show a lift simply because more people joined recently. Cohort analysis instead compares day-30 retention of the cohort that joined under the new flow against the day-30 retention of the cohort that joined under the old flow, giving a much more honest read on whether the change actually improved long-term behavior.