基于分割的注意力熵：在大视觉-语言模型中检测和缓解对象幻觉
Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models
AI总结：
本文提出基于分割的注意力熵(SAE)，通过语义分割量化视觉注意力不确定性，设计可靠性评分和注意力调整方法，有效缓解大视觉-语言模型中的对象幻觉问题。
摘要：
大型视觉-语言模型(LVLMs)在多种多模态任务中表现出色，但对象幻觉严重损害其可靠性。现有研究多聚焦文本模态，归因于过强的语言先验和不足的视觉支撑。相反，我们发现视觉模态中的异常注意力模式也能导致幻觉对象。基于此观察，我们提出基于分割的注意力熵(SAE)，利用语义分割在对象级语义空间中量化视觉注意力不确定性。基于SAE，我们进一步设计了幻觉检测的可靠性评分和SAE引导的注意力调整方法，在推理时修改视觉注意力以缓解幻觉。我们在公开基准和真实嵌入多模态场景中评估了我们的方法，实验结果表明，SAE在不增加额外训练成本的情况下显著减少了对象幻觉，从而实现了更可靠的LVLM驱动的感知和决策。
英文摘要：
Large Vision-Language Models (LVLMs) achieve strong performance on many multimodal tasks, but object hallucinations severely undermine their reliability. Most existing studies focus on the text modality, attributing hallucinations to overly strong language priors and insufficient visual grounding. In contrast, we observe that abnormal attention patterns within the visual modality can also give rise to hallucinated objects. Building on this observation, we propose Segmentation-based Attention Entropy (SAE), which leverages semantic segmentation to quantify visual attention uncertainty in a semantic space. Based on SAE, we further design a reliability score for hallucination detection and an SAE-guided attention adjustment method that modifies visual attention at inference time to mitigate hallucinations. We evaluate our approach on public benchmarks and in real embodied multimodal scenarios with quadruped robots. Experiments show that SAE reduces hallucinations without additional training, improving LVLM reliability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
