关于单模态模型与视觉-语言预训练模型对抗脆弱性的统一理解
A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models
AI总结：
针对视觉-语言预训练模型的安全漏洞，提出利用文本表示引导图像扰动的特征引导攻击（FGA）及结合文本攻击的FGA-T方法，有效实现了跨模态攻击并提升了黑盒迁移性，为评估VLP模型鲁棒性提供统一基线。
摘要：
随着视觉-语言预训练（VLP）模型展现出强大的多模态交互能力，神经网络的应用场景不再局限于单模态领域，而是扩展到了更复杂的多模态V+L下游任务。单模态模型的安全脆弱性已被广泛研究，而VLP模型的脆弱性仍具挑战性。我们注意到，在CV模型中，对图像的理解源于标注信息，而VLP模型旨在直接从原始文本中学习图像表示。受此差异启发，我们开发了特征引导攻击（FGA），这是一种利用文本表示来引导干净图像扰动的新方法，从而生成对抗图像。FGA与单模态领域许多先进攻击策略正交，便于将单模态领域的丰富研究成果直接应用于多模态场景。通过将文本攻击适当引入FGA，我们构建了带文本攻击的特征引导（FGA-T）。通过攻击两种模态的交互，FGA-T对VLP模型实现了更优的攻击效果。此外，结合数据增强和动量机制显著提升了FGA-T的黑盒可迁移性。我们的方法在各种数据集、下游任务以及黑盒和白盒设置中均展现出稳定有效的攻击能力，为探索VLP模型的鲁棒性提供了统一基线。
英文摘要：
With Vision-Language Pre-training (VLP) models demonstrating powerful multimodal interaction capabilities, the application scenarios of neural networks are no longer confined to unimodal domains but have expanded to more complex multimodal V+L downstream tasks. The security vulnerabilities of unimodal models have been extensively examined, whereas those of VLP models remain challenging. We note that in CV models, the understanding of images comes from annotated information, while VLP models are designed to learn image representations directly from raw text. Motivated by this discrepancy, we developed the Feature Guidance Attack (FGA), a novel method that uses text representations to direct the perturbation of clean images, resulting in the generation of adversarial images. FGA is orthogonal to many advanced attack strategies in the unimodal domain, facilitating the direct application of rich research findings from the unimodal to the multimodal scenario. By appropriately introducing text attack into FGA, we construct Feature Guidance with Text Attack (FGA-T). Through the interaction of attacking two modalities, FGA-T achieves superior attack effects against VLP models. Moreover, incorporating data augmentation and momentum mechanisms significantly improves the black-box transferability of FGA-T. Our method demonstrates stable and effective attack capabilities across various datasets, downstream tasks, and both black-box and white-box settings, offering a unified baseline for exploring the robustness of VLP models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
