基于文本驱动的攻击：提升编码器对抗迁移性以攻击大型视觉-语言模型
Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
AI总结：
提出文本驱动攻击（GDA），通过将扰动优化与文本接地证据对齐，并采用接地感知扰动分配和接地中心证据破坏策略，显著提升编码器攻击在黑盒大型视觉-语言模型上的迁移性。
摘要：
大型视觉-语言模型（LVLMs）在多模态任务中取得了令人印象深刻的性能，但它们对视觉输入的依赖使其面临对抗性威胁。编码器攻击通过仅通过视觉编码器生成扰动，为端到端优化提供了一种高效的替代方案。然而，现有的编码器攻击通常假设替代编码器与受害LVLM的视觉编码器相同或相似。在这项工作中，我们系统研究了它们在具有异构LVLM架构的更现实的黑盒部署中的迁移性。我们发现，模型特定的视觉证据在不同模型间不一致，而文本条件接地区域与标题相关证据更紧密相关，并提供了更稳定的迁移目标。然而，现有攻击与这些区域的对齐较弱且不足以破坏它们。受这些发现启发，我们提出了文本驱动攻击（GDA），它将扰动优化与文本接地证据对齐。GDA结合了接地感知扰动分配（将扰动预算集中在接地证据区域）和接地中心证据破坏（增强其全局和局部破坏）。在多种受害模型和任务上的实验表明，GDA在黑盒迁移中始终优于现有的编码器攻击。这些结果突显了文本接地证据在对抗迁移性中的核心作用，并激励了接地感知的鲁棒性评估和防御设计。
英文摘要：
Large vision-language models (LVLMs) have achieved impressive performance across multimodal tasks, but their reliance on visual inputs exposes them to adversarial threats. Encoder-based attacks provide an efficient alternative to end-to-end optimization by crafting perturbations through the vision encoder alone. However, existing encoder-based attacks often assume that the surrogate encoder is identical or similar to the victim LVLM's vision encoder. In this work, we present a systematic study of their transferability in more realistic black-box deployments with heterogeneous LVLM architectures. We find that model-specific visual evidence is inconsistent across models, whereas text-conditioned grounding regions are more closely tied to caption-relevant evidence and provide a more stable transfer target. However, existing attacks remain weakly aligned with and insufficiently disrupt these regions. Motivated by these findings, we propose Grounding-Driven Attack (GDA), which aligns perturbation optimization with text-grounded evidence. GDA combines Grounding-Aware Perturbation Allocation to concentrate perturbation budget on grounded evidence regions with Grounding-Centric Evidence Disruption to intensify their global and local disruption. Experiments across diverse victim models and tasks show that GDA consistently outperforms existing encoder-based attacks in black-box transfer. These results highlight the central role of text-grounded evidence in adversarial transferability and motivate grounding-aware robustness evaluation and defense design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
