从域偏移视角重新审视大型视觉-语言模型的后门攻击
Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift
AI总结：
本文从域偏移角度重新审视LVLM后门攻击，提出多模态归因后门攻击（MABA），通过域无关触发模式显著提升攻击泛化成功率，揭示当前评估局限及安全威胁。
摘要：
指令微调增强了大型视觉-语言模型（LVLMs）的能力，但由于其开放设计，也增加了它们对后门攻击的脆弱性。与以往在静态设置中的研究不同，本文探讨了在训练域与测试域不匹配的情况下，LVLM指令微调中的后门攻击。我们引入了一个新的评估维度——后门域泛化，用于评估在视觉和文本域偏移下的攻击鲁棒性。我们的发现揭示了两个见解：（1）当独特的触发模式独立于特定数据域或模型架构时，后门泛化能力会提高；（2）触发模式与干净语义区域之间存在竞争性交互，引导模型预测触发模式可以增强攻击的泛化能力。基于这些见解，我们提出了一种多模态归因后门攻击（MABA），利用归因解释将域无关的触发模式注入关键区域。在OpenFlamingo、Blip-2和Otter上的实验表明，MABA显著提高了攻击的泛化成功率36.4%，在0.2%的投毒率下达到了97%的成功率。这项研究揭示了当前评估的局限性，并强调了增强的后门泛化能力即使在没有测试数据访问的情况下，也对LVLMs构成安全威胁。
英文摘要：
Instruction tuning enhances large vision-language models (LVLMs) but increases their vulnerability to backdoor attacks due to their open design. Unlike prior studies in static settings, this paper explores backdoor attacks in LVLM instruction tuning across mismatched training and testing domains. We introduce a new evaluation dimension, backdoor domain generalization, to assess attack robustness under visual and text domain shifts. Our findings reveal two insights: (1) backdoor generalizability improves when distinctive trigger patterns are independent of specific data domains or model architectures, and (2) the competitive interaction between trigger patterns and clean semantic regions, where guiding the model to predict triggers enhances attack generalizability. Based on these insights, we propose a multimodal attribution backdoor attack (MABA) that injects domain-agnostic triggers into critical areas using attributional interpretation. Experiments with OpenFlamingo, Blip-2, and Otter show that MABA significantly boosts the attack success rate of generalization by 36.4%, achieving a 97% success rate at a 0.2% poisoning rate. This study reveals limitations in current evaluations and highlights how enhanced backdoor generalizability poses a security threat to LVLMs, even without test data access.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
