一个提示词就足以提升预训练视觉-语言模型的对抗鲁棒性
One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
AI总结：
本文提出APT，通过为冻结的CLIP等VLM学习仅含一个额外词的鲁棒文本提示，在15个数据集和多种数据稀疏设置下显著提升准确率与对抗鲁棒性。
摘要：
像CLIP这样的大型预训练视觉-语言模型（VLM）尽管具有出色的泛化能力，却极易受到对抗样本的影响。本工作从文本提示这一新视角研究VLM的对抗鲁棒性，而不是研究已被广泛探讨的模型权重（本工作中模型权重被冻结）。我们首先表明，对抗攻击和防御的有效性都对所使用的文本提示十分敏感。受此启发，我们提出了一种通过为VLM学习鲁棒文本提示来提升其对抗攻击抵御能力的方法。所提出的方法名为Adversarial Prompt Tuning（APT，对抗提示调优），不仅有效，而且在计算和数据方面都很高效。我们在15个数据集和4种数据稀疏方案（从1-shot到完整训练数据设置）上开展了大量实验，以证明APT相对于手工设计提示和其他最先进自适应方法的优越性。APT在分布内性能以及输入分布变化下和跨数据集的泛化方面均表现出优异能力。令人惊讶的是，只需在提示中添加一个学习得到的词，APT在epsilon=4/255时相比手工设计提示就能分别将准确率和鲁棒性平均显著提升+13%和+8.5%。在我们最有效的设置下，这一提升进一步增加到准确率+26.4%、鲁棒性+16.7%。代码可在https://github.com/TreeLLi/APT获取。
英文摘要：
Large pre-trained Vision-Language Models (VLMs) like CLIP, despite having remarkable generalization ability, are highly vulnerable to adversarial examples. This work studies the adversarial robustness of VLMs from the novel perspective of the text prompt instead of the extensively studied model weights (frozen in this work). We first show that the effectiveness of both adversarial attack and defense are sensitive to the used text prompt. Inspired by this, we propose a method to improve resilience to adversarial attacks by learning a robust text prompt for VLMs. The proposed method, named Adversarial Prompt Tuning (APT), is effective while being both computationally and data efficient. Extensive experiments are conducted across 15 datasets and 4 data sparsity schemes (from 1-shot to full training data settings) to show APT's superiority over hand-engineered prompts and other state-of-the-art adaption methods. APT demonstrated excellent abilities in terms of the in-distribution performance and the generalization under input distribution shift and across datasets. Surprisingly, by simply adding one learned word to the prompts, APT can significantly boost the accuracy and robustness (epsilon=4/255) over the hand-engineered prompts by +13% and +8.5% on average respectively. The improvement further increases, in our most effective setting, to +26.4% for accuracy and +16.7% for robustness. Code is available at https://github.com/TreeLLi/APT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
