什么使视觉语言模型具备鲁棒性？迈向视觉语言模型鲁棒性与准确性的协调
What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
AI总结：
本文研究了视觉语言模型鲁棒性与准确性的平衡问题，通过分析对抗性微调模型，发现鲁棒性主要集中在浅层网络，提出Adversarial Robustness Adaptation框架以提升鲁棒性与准确性。
摘要：
在视觉语言模型（VLMs）中实现对抗鲁棒性不可避免地会牺牲干净数据上的准确性，这是一个长期存在的挑战。本文重新审视这一权衡问题，通过分析对抗性微调模型，探讨鲁棒性机制如何在内部运作以及如何与干净准确性相互作用。我们的分析表明，对抗鲁棒性并非均匀分布在网络深度中，而是意外地主要集中在浅层网络中，由低频频谱偏差和输入不敏感的注意力模式驱动。同时，对深层网络的更新往往会损害干净准确性和鲁棒泛化能力。受这些见解启发，我们提出了Adversarial Robustness Adaptation（R-Adapt），一种简单而有效的框架，冻结所有预训练权重，并仅在初始层引入最小的洞察驱动的调整。这种设计在对抗鲁棒性和干净准确性之间实现了卓越的平衡。R-Adapt进一步支持无训练、模型引导和数据驱动的范式，为标准模型提供灵活的路径，使其无缝具备鲁棒性。在18个数据集和多样任务上的广泛评估显示了我们的最先进性能在各种攻击下。值得注意的是，R-Adapt能够高效泛化到大型视觉语言模型（如LLaVA和Qwen-VL）以增强其鲁棒性。我们的项目页面可在https://summu77.github.io/R-Adapt上找到。
英文摘要：
Achieving adversarial robustness in Vision-Language Models (VLMs) inevitably compromises accuracy on clean data, presenting a long-standing and challenging trade-off. In this work, we revisit this trade-off by investigating a fundamental question: What makes VLMs robust? Through a detailed analysis of adversarially fine-tuned models, we examine how robustness mechanisms function internally and how they interact with clean accuracy. Our analysis reveals that adversarial robustness is not uniformly distributed across network depth. Instead, unexpectedly, it is primarily localized within the shallow layers, driven by a low-frequency spectral bias and input-insensitive attention patterns. Meanwhile, updates to the deep layers tend to undermine both clean accuracy and robust generalization. Motivated by these insights, we propose Adversarial Robustness Adaptation (R-Adapt), a simple yet effective framework that freezes all pre-trained weights and introduces minimal, insight-driven adaptations only in the initial layers. This design achieves an exceptional balance between adversarial robustness and clean accuracy. R-Adapt further supports training-free, model-guided, and data-driven paradigms, offering flexible pathways to seamlessly equip standard models with robustness. Extensive evaluations on 18 datasets and diverse tasks demonstrate our state-of-the-art performance under various attacks. Notably, R-Adapt generalizes efficiently to large vision-language models (e.g., LLaVA and Qwen-VL) to enhance their robustness. Our project page is available at https://summu77.github.io/R-Adapt.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
