arXiv 2411.152442025-09-16cs.CVcs.AI

视觉语言模型的对抗提示蒸馏

Adversarial Prompt Distillation for Vision-Language Models

Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（上海智能信息处理实验室，计算机科学学院，复旦大学）；The Chinese University of Hong Kong, Shatin, Hong Kong（香港中文大学，沙田，香港）；The University of Hong Kong, Pokfulam, Hong Kong（香港大学，薄扶林，香港）

Lin Luo, Xin Wang, Bojia Zi, Shihao Zhao, Xingjun Ma, Yu-Gang Jiang

备注：: This work has been submitted to the IEEE for possible publication

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出双模态知识蒸馏框架APD，通过同时优化视觉和文本模态提示并从干净教师模型蒸馏知识，有效提升了视觉语言模型的对抗鲁棒性与干净准确率。

摘要：

大型预训练视觉语言模型（VLMs），如对比语言-图像预训练（CLIP），已被证明容易受到对抗攻击，引发了人们对将其部署在自动驾驶和医疗诊断等安全关键应用中的担忧。一种鲁棒化预训练VLMs的有前景方法是对抗提示微调（APT），它在提示微调过程中应用对抗训练。然而，现有的APT方法大多是单模态方法，仅为视觉或文本模态设计提示，限制了它们在鲁棒性或干净准确率上的有效性。在本工作中，我们提出对抗提示蒸馏（APD），一个双模态知识蒸馏框架，通过将APT与多模态知识转移集成来增强APT。APD同时优化视觉和文本模态的提示，并从干净的预训练教师CLIP模型中蒸馏知识。在多个基准数据集上的大量实验表明，我们的APD方法在对抗鲁棒性和干净准确率方面均优于当前最先进的APT方法。APD的有效性也验证了使用非鲁棒教师来提高微调VLMs的泛化能力和鲁棒性的可能性。

英文摘要：

Large pre-trained Vision-Language Models (VLMs) such as Contrastive Language-Image Pre-training (CLIP) have been shown to be susceptible to adversarial attacks, raising concerns about their deployment in safety-critical applications like autonomous driving and medical diagnosis. One promising approach for robustifying pre-trained VLMs is Adversarial Prompt Tuning (APT), which applies adversarial training during the process of prompt tuning. However, existing APT methods are mostly single-modal methods that design prompt(s) for only the visual or textual modality, limiting their effectiveness in either robustness or clean accuracy. In this work, we propose Adversarial Prompt Distillation (APD), a bimodal knowledge distillation framework that enhances APT by integrating it with multi-modal knowledge transfer. APD optimizes prompts for both visual and textual modalities while distilling knowledge from a clean pre-trained teacher CLIP model. Extensive experiments on multiple benchmark datasets demonstrate the superiority of our APD method over the current state-of-the-art APT methods in terms of both adversarial robustness and clean accuracy. The effectiveness of APD also validates the possibility of using a non-robust teacher to improve the generalization and robustness of fine-tuned VLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。