评估与分析大型视觉-语言模型中的关系幻觉
Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models
AI总结：
本研究提出R-Bench基准，系统评估大型视觉-语言模型中的关系幻觉，识别三种共现类型，并发现模型过度依赖常识而忽视视觉内容。
摘要：
幻觉问题在现有的大型视觉-语言模型（LVLMs）中是一个普遍关注的问题。以往的工作主要集中于研究物体幻觉，这类幻觉可以通过引入物体检测器来轻松缓解。然而，这些工作忽视了物体间关系的幻觉，而关系幻觉对于视觉理解至关重要。在本工作中，我们引入了R-Bench，一个用于评估视觉关系幻觉的新型基准。R-Bench包含图像级问题，聚焦于关系的存在性，以及实例级问题，用于评估局部视觉理解。我们识别出三种导致幻觉的关系共现类型：关系-关系、主体-关系和关系-物体。视觉指令调优数据集的长期尾部分布显著影响LVLMs对视觉关系的理解。此外，我们的分析揭示，当前的LVLMs倾向于忽视视觉内容，过度依赖大型语言模型的常识知识。它们还在基于上下文信息推理空间关系方面存在困难。
英文摘要：
The issue of hallucinations is a prevalent concern in existing Large Vision-Language Models (LVLMs). Previous efforts have primarily focused on investigating object hallucinations, which can be easily alleviated by introducing object detectors. However, these efforts neglect hallucinations in inter-object relationships, which is essential for visual comprehension. In this work, we introduce R-Bench, a novel benchmark for evaluating Vision Relationship Hallucination. R-Bench features image-level questions that focus on the existence of relationships and instance-level questions that assess local visual comprehension. We identify three types of relationship co-occurrences that lead to hallucinations: relationship-relationship, subject-relationship, and relationship-object. The visual instruction tuning dataset's long-tail distribution significantly impacts LVLMs' understanding of visual relationships. Furthermore, our analysis reveals that current LVLMs tend to disregard visual content and overly rely on the common sense knowledge of Large Language Models. They also struggle with reasoning about spatial relationships based on contextual information.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
