视觉编码器真的能解释物体幻觉吗？：通过简单细粒度CLIPScore缓解物体幻觉
Do Vision Encoders Truly Explain Object Hallucination?: Mitigating Object Hallucination via Simple Fine-Grained CLIPScore
AI总结：
本文提出F-CLIPScore，通过细粒度文本嵌入缓解LVLM中的物体幻觉问题，显著提升评估准确性。
摘要：
最近，大型视觉-语言模型（LVLM）在各种领域表现出色。然而，这些模型存在物体幻觉问题。在本文中，我们主要在判别、检索式评估设置（OHD-Caps）中研究物体幻觉，而不是在自由形式的标题生成中。本研究重新审视了之前关于此类幻觉原因的主张，即其根源在于视觉编码器的表示能力有限。我们的分析表明，视觉编码器的容量不一定是检测物体幻觉的主要限制因素。基于这一见解，我们提出了细粒度CLIPScore（F-CLIPScore），一种简单但有效的评估指标，通过在名词层面整合文本嵌入来增强物体层面的细粒度。在OHD-Caps基准上的评估表明，F-CLIPScore在准确性上显著优于传统CLIPScore，准确率提高了39.6%。我们进一步证明，基于F-CLIPScore的数据过滤可以减少LVLM中的物体幻觉（在对齐预训练后，POPE准确率提高4.9%）。我们的代码已公开在https://github.com/abzb1/f-clip
英文摘要：
Recently, Large Vision-Language Models (LVLMs) show remarkable performance across various domains. However, these models suffer from object hallucination. In this work, we study object hallucination primarily in a discriminative, retrieval-style evaluation setting (OHD-Caps), rather than in free-form caption generation. This study revisits the previous claim that the cause of such hallucinations lies in the limited representational capacity of the vision encoder. Our analysis implies that the capacity of the vision encoder is not necessarily a major limiting factor in detecting object hallucination. Based on this insight, we propose Fine-grained CLIPScore (F-CLIPScore), a simple yet effective evaluation metric that enhances object-level granularity by incorporating text embeddings at the noun level. Evaluations on the OHD-Caps benchmark show that F-CLIPScore significantly outperforms conventional CLIPScore in accuracy by a large margin of 39.6% without additional training. We further demonstrate that F-CLIPScore-based data filtering reduces object hallucination in LVLM (4.9% in POPE accuracy after alignment pretraining). Our code is publicly available at https://github.com/abzb1/f-clip
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
