CLIPure：基于CLIP的潜在空间纯化用于对抗鲁棒零样本分类
CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification
AI总结：
该研究提出CLIPure，一种基于CLIP多模态潜在空间的对抗鲁棒零样本分类纯化方法，含两个变体，在多个数据集上大幅提升了对抗鲁棒性，其中CLIPure-Cos不依赖生成模型且防御效率更高。
摘要：
本文旨在构建一个对抗鲁棒的零样本图像分类器，研究以CLIP为基础，CLIP是一种视觉-语言预训练编码器模型，可通过将图像与文本提示“一张<类别名称>的照片”匹配来执行零样本分类。本文选择纯化方法，因为它无需针对特定攻击类型进行对抗训练，因此可应对任何可预见的攻击。我们通过双向随机微分方程（SDE）将纯化风险公式化为：纯化过程（对对抗样本去噪）的联合分布与攻击过程（向良性样本添加扰动）的联合分布之间的KL散度。最终推导结果启发我们探索CLIP多模态潜在空间中的纯化。我们为CLIPure方法提出两个变体：CLIPure-Diff使用DaLLE-2中的DiffusionPrior模块对图像潜在向量的似然进行建模（对CLIP潜在向量的生成过程建模）；CLIPure-Cos使用图像嵌入与“一张<类别名称>的照片”之间的余弦相似度对似然进行建模。据我们所知，CLIPure是首个在多模态潜在空间中开展的纯化方法，且CLIPure-Cos是首个不基于生成模型的纯化方法，这大幅提升了防御效率。我们在CIFAR-10、ImageNet以及13个过往基于CLIP的防御方法用于评估零样本分类鲁棒性的数据集上开展了大量实验。结果显示，CLIPure大幅提升了当前最优（SOTA）鲁棒性，例如在CIFAR-10上从71.7%提升至91.1%，在ImageNet上从59.6%提升至72.6%，在这13个数据集上的平均鲁棒性较过往SOTA实现了108%的相对提升。代码可在指定URL获取。
英文摘要：
In this paper, we aim to build an adversarially robust zero-shot image classifier. We ground our work on CLIP, a vision-language pre-trained encoder model that can perform zero-shot classification by matching an image with text prompts ``a photo of a <class-name>.''. Purification is the path we choose since it does not require adversarial training on specific attack types and thus can cope with any foreseen attacks. We then formulate purification risk as the KL divergence between the joint distributions of the purification process of denoising the adversarial samples and the attack process of adding perturbations to benign samples, through bidirectional Stochastic Differential Equations (SDEs). The final derived results inspire us to explore purification in the multi-modal latent space of CLIP. We propose two variants for our CLIPure approach: CLIPure-Diff which models the likelihood of images' latent vectors with the DiffusionPrior module in DaLLE-2 (modeling the generation process of CLIP's latent vectors), and CLIPure-Cos which models the likelihood with the cosine similarity between the embeddings of an image and ``a photo of a.''. As far as we know, CLIPure is the first purification method in multi-modal latent space and CLIPure-Cos is the first purification method that is not based on generative models, which substantially improves defense efficiency. We conducted extensive experiments on CIFAR-10, ImageNet, and 13 datasets that previous CLIP-based defense methods used for evaluating zero-shot classification robustness. Results show that CLIPure boosts the SOTA robustness by a large margin, e.g., from 71.7% to 91.1% on CIFAR10, from 59.6% to 72.6% on ImageNet, and 108% relative improvements of average robustness on the 13 datasets over previous SOTA. The code is available at https://github.com/TMLResearchGroup-CAS/CLIPure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
