通过同心因果注意力缓解物体幻觉
Mitigating Object Hallucination via Concentric Causal Attention
AI总结：
本文提出同心因果注意力（CCA）位置对齐策略，通过缩小视觉与指令token的相对距离来缓解RoPE长距离衰减导致的物体幻觉，在多个基准上大幅超越现有方法。
摘要：
近期的大型视觉语言模型（LVLMs）在多模态查询下展现出卓越的零样本对话与推理能力。然而，它们存在物体幻觉问题，即LVLMs倾向于生成与图像输入在事实上不一致的文本响应。我们的初步研究表明，物体幻觉与旋转位置编码（RoPE）密切相关，RoPE是现有LVLMs中广泛采用的位置依赖建模设计。由于RoPE中的长距离衰减特性，当相关视觉线索在多模态输入序列中远离指令token时，LVLMs更容易产生幻觉。此外，我们观察到在多模态对齐过程中反转视觉token的序列顺序时也会产生类似效应。我们的测试表明，RoPE中的长距离衰减给LVLMs在长距离上捕捉视觉-指令交互带来了挑战。我们提出同心因果注意力（CCA），这是一种简单而有效的位置对齐策略，通过自然地缩小视觉token与指令token之间的相对距离，减轻RoPE长距离衰减对LVLMs的影响。借助CCA，视觉token能够更好地与指令token交互，从而增强模型的感知能力并缓解物体幻觉。无需额外复杂设计，我们的位置对齐方法在多个物体幻觉基准上以大幅优势超越了现有的幻觉缓解策略。
英文摘要：
Recent Large Vision Language Models (LVLMs) present remarkable zero-shot conversational and reasoning capabilities given multimodal queries. Nevertheless, they suffer from object hallucination, a phenomenon where LVLMs are prone to generate textual responses not factually aligned with image inputs. Our pilot study reveals that object hallucination is closely tied with Rotary Position Encoding (RoPE), a widely adopted positional dependency modeling design in existing LVLMs. Due to the long-term decay in RoPE, LVLMs tend to hallucinate more when relevant visual cues are distant from instruction tokens in the multimodal input sequence. Additionally, we observe a similar effect when reversing the sequential order of visual tokens during multimodal alignment. Our tests indicate that long-term decay in RoPE poses challenges to LVLMs while capturing visual-instruction interactions across long distances. We propose Concentric Causal Attention (CCA), a simple yet effective positional alignment strategy that mitigates the impact of RoPE long-term decay in LVLMs by naturally reducing relative distance between visual and instruction tokens. With CCA, visual tokens can better interact with instruction tokens, thereby enhancing model's perception capability and alleviating object hallucination. Without bells and whistles, our positional alignment method surpasses existing hallucination mitigation strategies by large margins on multiple object hallucination benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
