对抗过度依赖陷阱：通过自验证框架缓解LVLMs中的物体幻觉
Countering the Over-Reliance Trap: Mitigating Object Hallucination for LVLMs via a Self-Validation Framework
AI总结：
本文提出自验证框架，通过验证生成描述中物体存在的可信度，缓解LVLMs中的物体幻觉问题，在图像描述任务中取得显著改进。
摘要：
尽管在大视觉语言模型（LVLMs）中取得了进展，物体幻觉仍然是图像描述任务中的关键问题，其中模型生成不存在物体的描述，影响其可靠性。先前工作将此归因于LVLMs对语言先验的过度依赖，并尝试通过logits校准来缓解，但缺乏对过度依赖的深入分析。为了更深入地理解过度依赖，我们进行了一系列初步实验，表明随着生成长度的增加，LVLMs对语言先验的依赖导致幻觉物体标记的概率膨胀，从而加剧物体幻觉。为解决此问题，我们提出语言先验自由验证，使LVLMs能够忠实验证物体存在的置信度。基于此，我们提出一种新的训练自由的自验证框架，以对抗过度依赖陷阱。它首先验证采样候选描述中物体的存在，并通过描述选择或聚合进一步缓解物体幻觉。实验结果表明，我们的框架在图像描述任务中显著缓解了物体幻觉（例如，与LLaVA-v1.5-7B相比，CHAIRI指标提升了65.6%），超越了先前的SOTA方法。这一结果凸显了一条新的路径，通过解锁LVLMs自身内在潜力来缓解幻觉。
英文摘要：
Despite progress in Large Vision Language Models (LVLMs), object hallucination remains a critical issue in image captioning task, where models generate descriptions of non-existent objects, compromising their reliability. Previous work attributes this to LVLMs' over-reliance on language priors and attempts to mitigate it through logits calibration. However, they still lack a thorough analysis of the over-reliance. To gain a deeper understanding of over-reliance, we conduct a series of preliminary experiments, indicating that as the generation length increases, LVLMs' over-reliance on language priors leads to inflated probability of hallucinated object tokens, consequently exacerbating object hallucination. To circumvent this issue, we propose Language-Prior-Free Verification to enable LVLMs to faithfully verify the confidence of object existence. Based on this, we propose a novel training-free Self-Validation Framework to counter the over-reliance trap. It first validates objects' existence in sampled candidate captions and further mitigates object hallucination via caption selection or aggregation. Experiment results demonstrate that our framework mitigates object hallucination significantly in image captioning task (e.g., 65.6% improvement on CHAIRI metric with LLaVA-v1.5-7B), surpassing the previous SOTA methods. This result highlights a novel path towards mitigating hallucination by unlocking the inherent potential within LVLMs themselves.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
