Controlling Action-Head Co-Adaptation to VLM Representation Anisotropy
| Source: arXiv AI
Tags: AdaIE, Vision-Language-Action Models, RoboCasa, RoboTwin 2.0, Diffusion Policy
A study of vision-language-action models finds action heads over-focus on high-variance VLM dimensions, and a parameter-free fix called AdaIE adds an average 19 points of success across five real-world manipulation tasks.
Details
Robot policies increasingly pair a pretrained vision-language model with a downstream action model, but how that head adapts to the geometry of pretrained representations has been little studied. The authors find the action model progressively concentrates its sensitivity on high-variance dimensions of VLM outputs, and that counteracting this helps when representations and tasks are mismatched. Their Adaptive Interface Equalizer (AdaIE) reweights pretrained representations according to evolving downstream sensitivity during training, with no extra parameters. It improved success by up to 5.28 points on RoboCasa-GR1 and 1.68 points on RoboTwin 2.0, averaged 19 points across five real-world tasks, and also helped diffusion policies and world models.