Why GPT-Style Models Do Not Directly Transfer to Symbolic Music: Compression in the Wrong Coordinate System
| Source: arXiv AI
Tags: symbolic music, tokenization, generative models, music generation, arXiv
GPT-style tokenization fails for symbolic music not because musical tokens lack reusability, but because music lacks a coordinate system where recurring patterns form stable, predictively compressible distributions — the actual source of language tokenization success.
Details
Yi Wang proposes the Effectiveness-Losslessness Framework to explain why language model tokenization succeeds but does not transfer to symbolic music. The argument: GPT performance derives not from reusable token combinations alone, but from operating in a coordinate system where recurring regularities produce stable conditional distributions — effective compression. Standard music tokenizations fail to construct this coordinate system. The framework defines two key boundaries: the Fact-Token Boundary (decoupling and denesting expose predictive regularities) and the Token-State Boundary (tokenization stops before context-dependent relations are fixed, leaving computation to model states). Controlled symbolic music experiments validate both boundaries. Key result: sequence compaction alone — making longer, more reusable tokens — does not improve predictive compression. Preserving contextual freedom allows higher-order musical organization to emerge without explicit structural labels. The practical implication for music generation researchers: building models that genuinely leverage language architecture success requires discovering music-specific coordinate systems, not adapting tokenization schemes. No new model or benchmark accompanies the paper; this is a theoretical contribution.