PianoBench: Evaluating Musical Perception and Reasoning in Language Models
| Source: arXiv AI
Tags: PianoBench, audio-language models, Qwen 3.5, music understanding, chain of thought, multimodal benchmark
PianoBench shows open-weight audio-language models top out at 52.8% on musical reasoning from audio, while frontier models reach 93.2% when given music as text transcripts, exposing a perception gap.
Details
PianoBench tests whether language models can perceive and reason about piano performances across matched audio and symbolic inputs. It comes from a new dataset of more than 20,000 hours of interleaved music transcripts and spoken commentary from instructional piano videos. Open-weight audio-language models score at most 52.8% and only modestly beat a no-audio control. Frontier models with chain of thought reach up to 93.2% when given music transcripts as text, suggesting the bottleneck is hearing musical features rather than reasoning about them. Mid-training on interleaved musical instruction raised Qwen 3.5 9B Base's zero-shot accuracy with transcripts from 42.8% to 56.0%. The authors argue instructional videos are an overlooked source of multimodal supervision for music.