Diffusion Models and Concept Formation

Details

This paper, accepted as an oral at Advances of Cognitive Systems 2026, draws a formal correspondence between diffusion models and Cobweb, a cognitive model of concept formation built in the 1980s. Both systems build hierarchical probabilistic structures over data: Cobweb by incrementally constructing a discrete concept tree, and diffusion models by implicitly encoding a continuous hierarchy through the noise schedule. The key insight is that the noisy marginals of a diffusion process are Gaussian smoothings of the data distribution, and the modes of these marginals correspond to a hierarchy of prototypes matching a Cobweb tree. The researchers identify four shared properties: both are hierarchical density models, hierarchical-Bayesian models with Gaussian prototypes, treat categorization as score-following that reduces uncertainty, and both exhibit a 'basic level' of categorization. They locate the basic level in a diffusion model at an intermediate noise level, validated on MNIST and Fashion-MNIST. The work reframes diffusion as a cognitive model of concept formation—an interdisciplinary finding connecting machine learning to cognitive science. The practical impact is limited for practitioners, but the theoretical framing could influence how researchers think about what representations diffusion models actually learn.