幻觉图像令牌：一种用于检测和缓解LVLM中对象幻觉的免训练EAZY方法
Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMs
AI总结：
本文发现LVLM中仅1.5%的高注意力图像令牌是对象幻觉的主要来源，并提出免训练方法EAZY，通过将这些幻觉令牌置零实现无监督幻觉检测与缓解，检测性能较先前方法提升15%，且可无缝适配多种LVLM架构。
摘要：
尽管大型视觉语言模型（LVLMs）具有显著潜力，但它们仍面临对象幻觉的挑战，即生成的输出错误地包含了实际上并不存在的对象。虽然大多数工作侧重于在语言模型主干内解决此问题，但我们的工作将重点转移到图像输入源，研究特定图像令牌如何导致幻觉。我们的分析揭示了一个引人注目的发现：具有高注意力分数的一小部分图像令牌是对象幻觉的主要驱动因素。通过移除这些幻觉图像令牌（仅占所有图像令牌的1.5%），可以有效缓解该问题。这一发现在不同模型和数据集上均保持一致。基于这一见解，我们提出了EAZY，一种新颖的免训练方法，通过将幻觉图像令牌置零来自动识别并消除幻觉。我们利用EAZY进行无监督对象幻觉检测，与先前方法相比实现了15%的改进。此外，EAZY在缓解幻觉方面表现出显著的有效性，同时保持模型效用并无缝适应各种LVLM架构。
英文摘要：
Despite their remarkable potential, Large Vision-Language Models (LVLMs) still face challenges with object hallucination, a problem where their generated outputs mistakenly incorporate objects that do not actually exist. Although most works focus on addressing this issue within the language-model backbone, our work shifts the focus to the image input source, investigating how specific image tokens contribute to hallucinations. Our analysis reveals a striking finding: a small subset of image tokens with high attention scores are the primary drivers of object hallucination. By removing these hallucinatory image tokens (only 1.5% of all image tokens), the issue can be effectively mitigated. This finding holds consistently across different models and datasets. Building on this insight, we introduce EAZY, a novel, training-free method that automatically identifies and Eliminates hAllucinations by Zeroing out hallucinatorY image tokens. We utilize EAZY for unsupervised object hallucination detection, achieving 15% improvement compared to previous methods. Additionally, EAZY demonstrates remarkable effectiveness in mitigating hallucinations while preserving model utility and seamlessly adapting to various LVLM architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
