微调如你预训练：提升视觉语言模型零样本对抗鲁棒性
Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models
AI总结：
本文提出AdvFLYP方法，通过遵循CLIP预训练过程的训练配方，利用网络上收集的图像-文本对生成对抗样本，并通过对比损失匹配文本，提升视觉语言模型的零样本对抗鲁棒性。
摘要：
尽管视觉语言模型如CLIP在零样本能力方面表现出色，但它们对对抗攻击却十分脆弱。为增强其对抗鲁棒性，近期研究通过在代理数据集（如ImageNet）上对CLIP的预训练视觉编码器进行对抗样本微调，以对齐对抗图像与正确类标签。然而，这些方法忽略了训练数据分布和学习目标的重要性，导致零样本能力下降和鲁棒性在不同领域和数据集间的迁移性受限。在本文中，我们提出了一种简单而有效的范式AdvFLYP，其在进行对抗微调时遵循CLIP预训练过程的训练配方。具体而言，AdvFLYP利用从网络上收集的图像-文本对生成对抗图像，并通过对比损失匹配其对应的文本。为减轻噪声网络图像的对抗图像嵌入扭曲，我们进一步提出通过惩罚对抗图像特征的偏差来正则化AdvFLYP。我们展示了logit级和特征级的正则化项分别有助于鲁棒性和清洁准确率。在14个跨不同领域的下游数据集上的广泛实验显示，我们的范式优于主流方法。我们的代码和模型权重已发布在https://github.com/Sxing2/AdvFLYP。
英文摘要：
Despite their impressive zero-shot abilities, vision-language models such as CLIP have been shown to be susceptible to adversarial attacks. To enhance its adversarial robustness, recent studies finetune the pretrained vision encoder of CLIP with adversarial examples on a proxy dataset such as ImageNet by aligning adversarial images with correct class labels. However, these methods overlook the important roles of training data distributions and learning objectives, resulting in reduced zero-shot capabilities and limited transferability of robustness across domains and datasets. In this work, we propose a simple yet effective paradigm AdvFLYP, which follows the training recipe of CLIP's pretraining process when performing adversarial finetuning to the model. Specifically, AdvFLYP finetunes CLIP with adversarial images created based on image-text pairs collected from the web, and match them with their corresponding texts via a contrastive loss. To alleviate distortion of adversarial image embeddings of noisy web images, we further propose to regularise AdvFLYP by penalising deviation of adversarial image features. We show that logit- and feature-level regularisation terms benefit robustness and clean accuracy, respectively. Extensive experiments on 14 downstream datasets spanning various domains show the superiority of our paradigm over mainstream practices. Our code and model weights are released at https://github.com/Sxing2/AdvFLYP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
