Your AI Adoption Lift Is a Selection Effect

Details

When a company ships an opt-in AI assistant and compares adopters vs. non-adopters, the resulting retention lift almost always reflects who opted in, not what the feature did. Adopting an AI assistant requires an administrator who notices the launch, enables it, trains the team, integrates it into workflows, and sustains usage — every step is a signal of organizational readiness, which predicts retention independently. The feature is riding on top of that readiness, not creating it.\n\nThe author distinguishes three causal quantities that product decks routinely conflate: the effect on adopters specifically, the effect if all eligible accounts had adopted, and the marginal effect for accounts near the eligibility boundary. These are not the same number, and which one a business question requires determines the right methodology. Confusing them can send product roadmaps in the wrong direction.\n\nThe core recommendation is to stop modeling customer choice harder — propensity scores and covariate adjustment cannot remove bias driven by unobserved organizational readiness. Instead, exploit variation customers did not choose: the eligibility rule that determined which accounts could access the feature. This enables a regression discontinuity design, turning a business constraint into a natural experiment.\n\nThis is a rigorous practitioner guide aimed at data scientists inside product organizations. It applies directly to any team shipping AI features with opt-in adoption and trying to justify continued investment or measure true impact.