揭示视觉-语言模型的脆弱性：通过纹理约束扰动和跨模态优化的多模态对抗协同
Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
AI总结：
提出多模态对抗协同框架，通过纹理约束的通用对抗扰动和可学习的文本提示扰动，在黑盒设置下联合优化，揭示视觉-语言模型在多模态攻击下的脆弱性。
摘要：
大型视觉-语言模型（LVLMs）通过整合视觉和文本输入，在图像描述和视觉问答等任务中表现出色，改变了多模态理解。然而，它们对抗攻击的鲁棒性，特别是利用两种模态的攻击，仍未被充分探索，这给自动驾驶和内容审核等关键应用带来了风险。现有攻击集中于单一模态或需要不切实际的白盒访问，限制了其现实相关性。在本文中，我们引入了多模态对抗协同（MMAS），这是一个开创性的框架，用于针对LVLMs构建通用的黑盒多模态攻击。MMAS同时生成纹理尺度约束的通用对抗扰动用于图像，以及可学习的提示扰动用于文本，仅通过模型查询进行联合优化。图像扰动利用基于小波的纹理约束确保在各种视觉输入中的不可感知性和鲁棒性。文本扰动在嵌入空间中受L范数约束，在保持语义连贯性的同时将输出导向目标。一种新颖的跨模态正则化项对齐扰动的梯度方向，增强了它们在任务和模型间的协同影响和可迁移性。大量实验表明，我们提出的攻击在主流LVLMs上具有强大的通用对抗能力。
英文摘要：
Large Vision-Language Models (LVLMs) have transformed multi-modal understanding, excelling in tasks like image captioning and visual question answering by integrating visual and textual inputs. However, their robustness against adversarial attacks, particularly those exploiting both modalities, remains underexplored, posing risks to critical applications like autonomous driving and content moderation. Existing attacks focus on single modalities or require impractical white-box access, limiting their real-world relevance. In this paper, we introduce Multi-Modal Adversarial Synergy, a groundbreaking framework that crafts universal, black-box multi-modal attacks against LVLMs. MMAS simultaneously generates a texture scale-constrained universal adversarial perturbation for images and a learnable prompt perturbation for text, optimized jointly using only model queries. The image perturbation leverages wavelet-based texture constraints to ensure imperceptibility and robustness across diverse visual inputs. The text perturbation, constrained by an L-norm in the embedding space, maintains semantic coherence while steering outputs toward a target. A novel cross-modal regularization term aligns the perturbations' gradient directions, enhancing their synergistic impact and transferability across tasks and models. Extensive experiments show the strong universal adversarial capabilities of our proposed attack with prevalent LVLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
