FIHA：基于Davidson场景图的视觉语言模型自主幻觉评估
FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphs
AI总结：
本文提出FIHA方法，通过免标注、免大语言模型的方式自动生成问答对，并利用Davidson场景图建模幻觉依赖关系，构建FIHA-v1基准以全面评估视觉语言模型的细粒度幻觉问题。
摘要：
大型视觉语言模型（LVLMs）的快速发展往往伴随着广泛的幻觉问题，这使得低成本且全面的评估变得越来越重要。当前的方法主要依赖昂贵的标注，并且在评估所有方面（如关系、属性以及各方面之间的依赖关系）时并不全面。因此，我们提出了FIHA（LVLMs中的自主细粒度幻觉评估），该方法能够在无需大语言模型且无需标注的情况下评估LVLMs的幻觉，并对不同类型幻觉之间的依赖关系进行建模。FIHA能够以极低的成本在任意图像数据集上生成问答对，从而支持从图像和文本描述两个角度进行幻觉评估。基于该方法，我们引入了一个名为FIHA-v1的基准，该基准包含来自MSCOCO和Foggy数据集中各种图像上的多样化问题。此外，我们使用Davidson场景图（DSG）来组织问答对之间的结构，从而提高评估的可靠性。我们使用FIHA-v1对代表性模型进行了评估，揭示了它们的局限性与挑战。我们已发布代码和数据。
英文摘要：
The rapid development of Large Vision-Language Models (LVLMs) often comes with widespread hallucination issues, making cost-effective and comprehensive assessments increasingly vital. Current approaches mainly rely on costly annotations and are not comprehensive -- in terms of evaluating all aspects such as relations, attributes, and dependencies between aspects. Therefore, we introduce the FIHA (autonomous Fine-graIned Hallucination evAluation evaluation in LVLMs), which could access hallucination LVLMs in the LLM-free and annotation-free way and model the dependency between different types of hallucinations. FIHA can generate Q&A pairs on any image dataset at minimal cost, enabling hallucination assessment from both image and caption. Based on this approach, we introduce a benchmark called FIHA-v1, which consists of diverse questions on various images from MSCOCO and Foggy. Furthermore, we use the Davidson Scene Graph (DSG) to organize the structure among Q&A pairs, in which we can increase the reliability of the evaluation. We evaluate representative models using FIHA-v1, highlighting their limitations and challenges. We released our code and data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
