arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.05491cs.CVcs.CR
收藏

一次扰动就足够:针对视觉-语言预训练模型生成通用对抗扰动

One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models

Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);Harbin Institute of Technology(哈尔滨工业大学)

Hao Fang, Jiawei Kong, Wenbo Yu, Bin Chen, Jiawei Li, Hao Wu, Shutao Xia, Ke Xu

AI总结:

本文提出C-PGC方法,通过对比学习生成通用对抗扰动,攻击视觉-语言预训练模型的多模态对齐能力。

摘要:

视觉-语言预训练(VLP)模型通过充分利用多模态对齐,在许多应用中展现了前所未有的能力。然而,先前的研究表明它们容易受到恶意构造的对抗样本的攻击。尽管最近取得了成功,但这些方法通常是实例特定的,并且需要为每个输入样本生成扰动。在本文中,我们揭示VLP模型也容易受到实例无关的通用对抗扰动(UAP)的攻击。具体来说,我们设计了一种新颖的对比训练扰动生成器与跨模态条件(C-PGC)以实现攻击。鉴于关键的多模态对齐是通过先进的对比学习技术实现的,我们设计将这种强大的武器用于自身,即使用恶意版本的对比学习来训练C-PGC,基于我们精心设计的正负图像-文本对,以实质上破坏VLP模型学习的对齐关系。此外,C-PGC充分利用了视觉-语言(V+L)场景的特点,通过结合单模态和跨模态信息作为有效的指导。广泛的实验表明,C-PGC成功地迫使对抗样本在VLP模型的特征空间中远离其原始区域,从而在各种受害者模型和V+L任务中实质上增强了攻击。GitHub仓库可在https://github.com/ffhibnese/CPGC_VLP_Universal_Attacks上获取。

英文摘要:

Vision-Language Pre-training (VLP) models have exhibited unprecedented capability in many applications by taking full advantage of the multimodal alignment. However, previous studies have shown they are vulnerable to maliciously crafted adversarial samples. Despite recent success, these methods are generally instance-specific and require generating perturbations for each input sample. In this paper, we reveal that VLP models are also vulnerable to the instance-agnostic universal adversarial perturbation (UAP). Specifically, we design a novel Contrastive-training Perturbation Generator with Cross-modal conditions (C-PGC) to achieve the attack. In light that the pivotal multimodal alignment is achieved through the advanced contrastive learning technique, we devise to turn this powerful weapon against themselves, i.e., employ a malicious version of contrastive learning to train the C-PGC based on our carefully crafted positive and negative image-text pairs for essentially destroying the alignment relationship learned by VLP models. Besides, C-PGC fully utilizes the characteristics of Vision-and-Language (V+L) scenarios by incorporating both unimodal and cross-modal information as effective guidance. Extensive experiments show that C-PGC successfully forces adversarial samples to move away from their original area in the VLP model's feature space, thus essentially enhancing attacks across various victim models and V+L tasks. The GitHub repository is available at https://github.com/ffhibnese/CPGC_VLP_Universal_Attacks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑