学习无需学习的内容：用于鲁棒视觉-语言模型的对抗性解耦提示调优

中文摘要 AI 辅助

尽管对抗性提示调优可高效增强视觉-语言模型的鲁棒性，但研究发现现有方法会加剧对已见类别的鲁棒泛化过拟合，导致随着训练推进，模型对未见类别的对抗样本性能迅速下降。经实证分析，该退化源于模型倾向于学习伪鲁棒特征（即不可泛化的捷径）。为缓解此问题，本文提出ADAPT（Adversarial Disentangled Prompt Tuning，对抗性解耦提示调优），遵循“学习无需学习的内容”理念的鲁棒提示调优框架。具体而言，ADAPT采用双提示机制，包含目标提示与一组诱饵提示；训练期间，诱饵提示被引导捕获多样化伪鲁棒特征，而目标提示被约束为在嵌入空间中与诱饵提示正交，以学习鲁棒特征。通过将鲁棒特征与伪鲁棒特征解耦，ADAPT可有效防止鲁棒泛化过拟合。进一步分析表明，正交损失可限制伪鲁棒特征的变化对未见类别的影响，从而提供测试误差保证。大量实验实证表明，ADAPT可显著提升目标提示对未见类别的鲁棒性。代码可在指定URL获取。

英文摘要

While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts). To mitigate this, we propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework following the philosophy of ``Learning What Not to Learn''. Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features. By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes. The code is available at https://github.com/cheny02/ADAPT-ACMMM2026.

发表机构

• Southern University of Science and Technology（南方科技大学）
• City University of Hong Kong（香港城市大学）
• Hong Kong University of Science and Technology（香港科技大学）

机构由 AI 辅助整理，请以论文原文为准。