使用鲁棒编码器保护视觉-语言模型免受越狱和对抗性攻击
Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks
AI总结：
针对大型视觉-语言模型易受对抗攻击和越狱攻击的问题，提出即插即用的Sim-CLIP+防御机制，通过孪生架构对抗微调CLIP视觉编码器，在保持高干净准确率的同时大幅提升攻击鲁棒性。
摘要：
在多模态大数据集上训练的大型视觉-语言模型（LVLM）凭借在视觉-语言任务中的出色表现极大推动了AI发展，但这类模型仍易受对抗性攻击，尤其是越狱攻击——这类攻击会绕过安全协议，导致模型生成误导性或有害响应。该脆弱性既源于大语言模型（LLM）的固有易感性，也来自视觉模态引入的扩大攻击面。我们提出Sim-CLIP+，这是一种新型防御机制，通过利用孪生架构对CLIP视觉编码器进行对抗性微调。该方法最大化扰动样本与干净样本之间的余弦相似度，从而提升对对抗性操控的抵御能力。Sim-CLIP+是即插即用的解决方案，可作为鲁棒视觉编码器无缝集成到现有LVLM架构中。与以往防御方法不同，我们的方法无需对LVLM进行结构修改，且计算开销极小。Sim-CLIP+在抵御基于梯度的对抗性攻击和多种越狱技术方面均展现出有效性。我们针对三种不同的越狱攻击策略评估了Sim-CLIP+，并使用标准下游数据集进行了干净样本评估，包括用于图像字幕生成的COCO和用于视觉问答的OKVQA。大量实验表明，Sim-CLIP+在保持高干净样本准确率的同时，大幅提升了对基于梯度的对抗性攻击和越狱技术的鲁棒性。我们的代码和鲁棒视觉编码器可在https://github.com/speedlab-git/Robust-Encoder-against-Jailbreak-attack.git获取。
英文摘要：
Large Vision-Language Models (LVLMs), trained on multimodal big datasets, have significantly advanced AI by excelling in vision-language tasks. However, these models remain vulnerable to adversarial attacks, particularly jailbreak attacks, which bypass safety protocols and cause the model to generate misleading or harmful responses. This vulnerability stems from both the inherent susceptibilities of LLMs and the expanded attack surface introduced by the visual modality. We propose Sim-CLIP+, a novel defense mechanism that adversarially fine-tunes the CLIP vision encoder by leveraging a Siamese architecture. This approach maximizes cosine similarity between perturbed and clean samples, facilitating resilience against adversarial manipulations. Sim-CLIP+ offers a plug-and-play solution, allowing seamless integration into existing LVLM architectures as a robust vision encoder. Unlike previous defenses, our method requires no structural modifications to the LVLM and incurs minimal computational overhead. Sim-CLIP+ demonstrates effectiveness against both gradient-based adversarial attacks and various jailbreak techniques. We evaluate Sim-CLIP+ against three distinct jailbreak attack strategies and perform clean evaluations using standard downstream datasets, including COCO for image captioning and OKVQA for visual question answering. Extensive experiments demonstrate that Sim-CLIP+ maintains high clean accuracy while substantially improving robustness against both gradient-based adversarial attacks and jailbreak techniques. Our code and robust vision encoders are available at https://github.com/speedlab-git/Robust-Encoder-against-Jailbreak-attack.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
