利用视觉编码器漏洞对大视觉-语言模型生成通用对抗扰动
Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models
AI总结：
本文研究了大视觉-语言模型在输入图像中对小对抗扰动的脆弱性，提出了一种针对视觉编码器内部组件的通用对抗扰动攻击框架，通过分析注意力机制识别关键脆弱组件，实现高效攻击。
摘要：
大视觉-语言模型（LVLMs）在多模态任务中表现出色，但对输入图像中的小对抗扰动高度敏感。现有攻击通常针对视觉编码器的最终输出嵌入，隐含地将编码器视为统一的攻击面，而对哪些内部组件最脆弱的系统性分析仍很少探讨。本文证明这种分析至关重要，因为LVLM视觉编码器的对抗脆弱性在结构上集中而非均匀分布。基于此，我们提出视觉编码器脆弱组件目标通用对抗扰动（VEV-UAP），一种任务无关且成本效益高的攻击框架。通过组件和层级分析注意力机制，我们发现中间层的价值组件是关键脆弱点，强烈影响下游语言模型行为。VEV-UAP选择性地针对这些组件生成单一通用扰动，不涉及文本输入或语言模型的优化过程。在多个LVLM和任务上的实验表明，VEV-UAP实现了最先进的攻击成功率，同时减少了计算开销。此外，单个VEV-UAP在共享相同视觉编码器的不同LVLM之间可迁移，即使配对不同的语言模型，使其成为可扩展鲁棒性评估的实用框架。
英文摘要：
Large Vision-Language Models (LVLMs) have achieved remarkable performance on multimodal tasks but remain highly vulnerable to small adversarial perturbations in input images. Existing attacks typically target the vision encoder's final output embeddings, implicitly treating the encoder as a uniform attack surface, while a systematic analysis of which internal components are most vulnerable has remained largely unexplored. We show such analysis is essential, as adversarial vulnerability in LVLM vision encoders is structurally concentrated rather than uniformly distributed. Building on this, we propose Vision Encoder Vulnerable-Component-Targeted Universal Adversarial Perturbation (VEV-UAP), a task-agnostic and cost-efficient attack framework. Through a component- and layer-wise analysis of attention mechanisms, we identify the value components in middle layers as critical vulnerabilities that strongly influence downstream language model behavior. VEV-UAP selectively targets these components to generate a single universal perturbation shared across images, without involving textual inputs or the language model during optimization. Experiments across multiple LVLMs and tasks show VEV-UAP achieves state-of-the-art attack success rates with reduced computational overhead. Moreover, a single VEV-UAP transfers across LVLMs sharing the same vision encoder, even when paired with different language models, making it a practical framework for scalable robustness evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
