眼中的幻象：仅利用注意力沉池对多模态大语言模型的幻觉攻击
Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink
AI总结：
本研究揭示多模态大语言模型指令微调中的固有漏洞，提出利用注意力沉池行为的新型幻觉攻击方法，可生成动态、高可迁移的视觉对抗输入，有效攻破黑盒模型及GPT-4o等商业API。
摘要：
多模态大语言模型（MLLMs）将视觉理解融入语言生成，正在彻底改变视觉语言应用。然而，这些模型常受幻觉问题困扰，即生成与视觉内容不符的不准确对象、属性和关系。在本研究中，我们深入探究MLLMs的内部注意力机制以揭示幻觉的潜在成因，暴露了指令微调过程中的固有漏洞。我们提出了一种针对MLLMs的新型幻觉攻击，利用注意力沉池行为，以极小的图文相关性触发幻觉内容，对关键下游应用构成重大威胁。与以往依赖固定模式的对抗方法不同，我们的方法生成动态、有效且高度可迁移的视觉对抗输入，且不牺牲模型响应质量。在6个知名MLLMs上的全面实验表明，即使具备广泛缓解机制，我们的攻击仍能有效攻破黑盒MLLMs，并在对抗GPT-4o和Gemini 1.5等前沿商业API时取得理想结果。代码可在https://huggingface.co/RachelHGF/Mirage-in-the-Eyes获取。
英文摘要：
Fusing visual understanding into language generation, Multi-modal Large Language Models (MLLMs) are revolutionizing visual-language applications. Yet, these models are often plagued by the hallucination problem, which involves generating inaccurate objects, attributes, and relationships that do not match the visual content. In this work, we delve into the internal attention mechanisms of MLLMs to reveal the underlying causes of hallucination, exposing the inherent vulnerabilities in the instruction-tuning process. We propose a novel hallucination attack against MLLMs that exploits attention sink behaviors to trigger hallucinated content with minimal image-text relevance, posing a significant threat to critical downstream applications. Distinguished from previous adversarial methods that rely on fixed patterns, our approach generates dynamic, effective, and highly transferable visual adversarial inputs, without sacrificing the quality of model responses. Comprehensive experiments on 6 prominent MLLMs demonstrate the efficacy of our attack in compromising black-box MLLMs even with extensive mitigating mechanisms, as well as the promising results against cutting-edge commercial APIs, such as GPT-4o and Gemini 1.5. Our code is available at https://huggingface.co/RachelHGF/Mirage-in-the-Eyes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
