双重视觉防御：对抗性预训练与指令微调用于提升视觉-语言模型鲁棒性
Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
AI总结：
本文提出“双重视觉防御”方法，通过大规模对抗性视觉-语言预训练和对抗性视觉指令微调，分别得到$Δ$CLIP和$Δ^2$LLaVA模型，显著提升了视觉-语言模型在图像分类、图像描述和视觉问答等任务上的对抗鲁棒性。
摘要：
本文研究了视觉-语言模型面对对抗性视觉扰动时的鲁棒性，并提出了一种新颖的“双重视觉防御”方法来增强这种鲁棒性。与以往采用对预训练CLIP模型进行轻量级对抗微调的方法不同，我们利用网络规模的数据从头开始进行大规模对抗性视觉-语言预训练。随后，我们通过引入对抗性视觉指令微调来进一步加强防御。每个阶段得到的模型，即$Δ$CLIP和$Δ^2$LLaVA，都展现出显著增强的零样本鲁棒性，并在视觉-语言模型的对抗防御方面树立了新的最优水平。例如，$Δ$CLIP在ImageNet-1k上的对抗鲁棒性比之前的最佳模型高出约20%。类似地，与现有技术相比，$Δ^2$LLaVA为图像描述任务带来了约30%的鲁棒性提升，为视觉问答任务带来了约20%的鲁棒性提升。此外，与基线模型相比，我们的模型展现出更强的零样本识别能力、更少的幻觉现象以及更优越的推理性能。我们的项目主页是https://doublevisualdefense.github.io/。
英文摘要：
This paper investigates the robustness of vision-language models against adversarial visual perturbations and introduces a novel ``double visual defense" to enhance this robustness. Unlike previous approaches that resort to lightweight adversarial fine-tuning of a pre-trained CLIP model, we perform large-scale adversarial vision-language pre-training from scratch using web-scale data. We then strengthen the defense by incorporating adversarial visual instruction tuning. The resulting models from each stage, $Δ$CLIP and $Δ^2$LLaVA, show substantially enhanced zero-shot robustness and set a new state-of-the-art in adversarial defense for vision-language models. For example, the adversarial robustness of $Δ$CLIP surpasses that of the previous best models on ImageNet-1k by ~20%. %For example, $Δ$CLIP surpasses the previous best models on ImageNet-1k by ~20% in terms of adversarial robustness. Similarly, compared to prior art, $Δ^2$LLaVA brings a ~30% robustness improvement to image captioning task and a ~20% robustness improvement to visual question answering task. Furthermore, our models exhibit stronger zero-shot recognition capability, fewer hallucinations, and superior reasoning performance compared to baselines. Our project page is https://doublevisualdefense.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
