Scaling Categorical Flow Maps
| Source: Apple ML Research
Tags: Apple, flow matching, diffusion language models, Categorical Flow Maps, inference efficiency, language models
Apple researchers scaled Categorical Flow Maps to 1.7B parameters trained on 2.1T tokens, demonstrating that discrete-data flow matching can generate competitive text in as few as 4 inference steps — the first credible scaling test of this architecture beyond 1B parameters.
Details
Apple's ML team published research scaling Categorical Flow Maps (CFMs) to 1.7B parameters trained on 2.1T tokens, addressing a key open question: can discrete-data flow matching architectures compete with autoregressive models at real scale? The team first trains a base flow model, then self-distills it into a CFM that generates text in as few as 4 inference steps while preserving near-data-level token entropy. Previous work on CFMs had only been evaluated below 1B parameters, leaving scalability unproven. Two additional contributions: a likelihood bound for CFMs in the semi-discrete setting (enabling standard LM benchmark scoring previously impossible for this architecture), and prescriptive guidance on loss weighting and time scheduling that other researchers will need when scaling these models. The work is foundational research rather than a product release. Non-autoregressive generation could eventually unlock significant inference speedups, but practical deployment implications remain distant. The result establishes a viable research trajectory for an architecture family that is gaining serious attention.