Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models

Details

Adversarial prompt tuning is a parameter-efficient approach to making vision-language models like CLIP robust against adversarial examples — only the prompt tokens are trained, not the full model. But the authors find that existing methods suffer from robust generalization overfitting: performance on seen training classes improves while robustness on unseen test classes degrades as training progresses. The root cause they identify is that models learn 'pseudo-robust features' — shortcuts that look like genuine robustness signals but do not generalize. ADAPT (Adversarial Disentangled Prompt Tuning) addresses this with a dual-prompt architecture: a 'target prompt' is trained to be orthogonal to a pool of 'decoy prompts' that are explicitly optimized to capture diverse pseudo-robust features. By forcing separation in the embedding space, ADAPT prevents the target prompt from overfitting on non-generalizable shortcuts. The authors provide a theoretical bound showing that the orthogonal constraint limits how much shifts in pseudo-robust features affect test error on unseen classes. Empirical results across multiple evaluations confirm that ADAPT substantially improves robustness on unseen classes. Code is available. This is relevant to practitioners deploying VLMs in safety-sensitive applications where distribution shift over categories is expected.