高质量文本，稳健视觉：语言在增强视觉语言模型视觉稳健性中的作用
Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models
AI总结：
研究针对视觉语言模型对抗攻击问题，提出质量文本引导的对抗微调方法QT-AFT，利用高质量字幕提升视觉编码器对抗鲁棒性，在多个零样本数据集上实现了最先进的零样本对抗鲁棒性和纯净准确率，并揭示了语言增强视觉鲁棒性的关键见解。
摘要：
防御预训练视觉语言模型（如CLIP）免受对抗攻击至关重要，因其广泛用于包括图像分类在内的各种零样本任务。现有对抗训练方法存在不足，监督式对抗训练依赖短文本生成对抗扰动导致过拟合，无监督式对抗训练缺乏语义指导。为此提出质量文本引导的对抗微调（QT-AFT），利用高质量字幕引导对抗示例远离图像中的各种语义，提升视觉编码器在对抗噪声下识别图像特征的能力，在16个零样本数据集上实现了零样本对抗鲁棒性和纯净准确率的提升，还揭示了语言在增强视觉鲁棒性方面的关键见解。
英文摘要：
Defending pre-trained vision-language models (VLMs), such as CLIP, against adversarial attacks is crucial, as these models are widely used in diverse zero-shot tasks, including image classification. However, existing adversarial training (AT) methods for robust fine-tuning largely overlook the role of language in enhancing visual robustness. Specifically, (1) supervised AT methods rely on short texts (e.g., class labels) to generate adversarial perturbations, leading to overfitting to object classes in the training data, and (2) unsupervised AT avoids this overfitting but remains suboptimal against practical text-guided adversarial attacks due to its lack of semantic guidance. To address these limitations, we propose Quality Text-guided Adversarial Fine-Tuning (QT-AFT), which leverages high-quality captions during training to guide adversarial examples away from diverse semantics present in images. This enables the visual encoder to robustly recognize a broader range of image features even under adversarial noise, thereby enhancing robustness across diverse downstream tasks. QT-AFT overcomes the key weaknesses of prior methods -- overfitting in supervised AT and lack of semantic awareness in unsupervised AT -- achieving state-of-the-art zero-shot adversarial robustness and clean accuracy, evaluated across 16 zero-shot datasets. Furthermore, our comprehensive study uncovers several key insights into the role of language in enhancing vision robustness; for example, describing object properties in addition to object names further enhances zero-shot robustness. Our findings point to an urgent direction for future work -- centering high-quality linguistic supervision in robust visual representation learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
