视觉描述接地减少LVLMs中的幻觉并提升推理能力
Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs
AI总结：
本文提出无需训练的VDGD方法，通过生成图像描述作为指令前缀并基于KL散度采样令牌，增强LVLMs的视觉感知与推理能力，减少幻觉，在多基准上性能提升2%-33%，并推出认知评估基准VaLLu。
摘要：
大型视觉语言模型（LVLMs）经常产生与事实信息不符的响应，这种现象被称为幻觉。虽然幻觉已被广泛研究，但其背后的确切原因仍未被充分探讨。在本文中，我们首先调查了LVLMs中幻觉的根本原因。我们的发现表明，现有的缓解技术主要减少了视觉识别提示（即那些需要对视觉元素进行简单描述的提示）的幻觉，但对于需要深思熟虑推理的认知提示却失败了。我们将核心问题确定为LVLMs缺乏真正的视觉感知：尽管它们能够准确识别视觉元素，但它们难以在输入提示的上下文中完全解释这些元素，并有效地将这种识别与其内部知识联系起来，而这对于推理至关重要。为了解决这一差距，我们引入了视觉描述接地解码（Visual Description Grounded Decoding, VDGD），这是一种简单、稳健且无需训练的方法，旨在增强LVLMs的视觉感知并改善推理能力。VDGD的工作原理是首先生成图像的详细描述，并将其作为前缀附加到指令中。在响应生成期间，根据令牌与描述的KL散度对令牌进行采样，偏好具有较低散度的候选者。在多个视觉推理基准和LVLMs上的实验结果表明，VDGD始终优于现有基线2%至33%。最后，我们引入了VaLLu，一个旨在全面评估LVLMs认知能力的基准。
英文摘要：
Large Vision-Language Models (LVLMs) often produce responses that misalign with factual information, a phenomenon known as hallucinations. While hallucinations are well-studied, the exact causes behind them remain underexplored. In this paper, we first investigate the root causes of hallucinations in LVLMs. Our findings reveal that existing mitigation techniques primarily reduce hallucinations for visual recognition prompts-those that require simple descriptions of visual elements-but fail for cognitive prompts that demand deliberate reasoning. We identify the core issue as a lack of true visual perception in LVLMs: although they can accurately recognize visual elements, they struggle to fully interpret these elements in the context of the input prompt and effectively link this recognition to their internal knowledge, which is critical for reasoning. To address this gap, we introduce Visual Description Grounded Decoding (VDGD), a simple, robust, and training-free method designed to enhance visual perception and improve reasoning capabilities in LVLMs. VDGD works by first generating a detailed description of the image and appending it as a prefix to the instruction. During response generation, tokens are sampled based on their KL divergence to the description, favoring candidates with lower divergence. Experimental results on multiple visual reasoning benchmarks and LVLMs demonstrate that VDGD consistently outperforms existing baselines 2% - 33%. Finally, we introduce VaLLu, a benchmark designed for comprehensive evaluation of the cognitive capabilities of LVLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
