用视觉为语言奠基：一种基于条件互信息校准的解码策略以减少LVLMs中的幻觉
Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
AI总结：
针对LVLMs解码时过度依赖语言先验导致的幻觉问题，提出基于条件逐点互信息校准的解码策略，联合建模视觉与文本标记贡献，通过双层优化和标记净化机制最大化互信息，在多个基准上显著减少幻觉并保持解码效率。
摘要：
大型视觉-语言模型（LVLMs）容易产生幻觉，即生成的回复在语义上看似合理，但与输入图像几乎没有或完全没有相关性。先前的研究表明，这一问题主要源于LVLMs在解码过程中过度依赖语言先验，而忽视了视觉信息。为缓解此问题，我们提出了一种新颖的基于条件逐点互信息（C-PMI）校准的解码策略，该策略自适应地增强生成文本与输入图像之间的相互依赖关系，从而减轻幻觉。与现有仅关注文本标记采样的方法不同，我们提出联合建模视觉和文本标记对C-PMI的贡献，将幻觉缓解表述为一个旨在最大化互信息的双层优化问题。为解决该问题，我们设计了一种标记净化机制，通过采样与给定图像保持最大相关性的文本标记，同时精炼与生成回复最相关的图像标记，来动态调节解码过程。在多个基准上的大量实验表明，所提方法在保持解码效率的同时显著减少了LVLMs中的幻觉。
英文摘要：
Large Vision-Language Models (LVLMs) are susceptible to hallucinations, where generated responses seem semantically plausible yet exhibit little or no relevance to the input image. Previous studies reveal that this issue primarily stems from LVLMs' over-reliance on language priors while disregarding the visual information during decoding. To alleviate this issue, we introduce a novel Conditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy, which adaptively strengthens the mutual dependency between generated texts and input images to mitigate hallucinations. Unlike existing methods solely focusing on text token sampling, we propose to jointly model the contributions of visual and textual tokens to C-PMI, formulating hallucination mitigation as a bi-level optimization problem aimed at maximizing mutual information. To solve it, we design a token purification mechanism that dynamically regulates the decoding process by sampling text tokens remaining maximally relevant to the given image, while simultaneously refining image tokens most pertinent to the generated response. Extensive experiments across various benchmarks reveal that the proposed method significantly reduces hallucinations in LVLMs while preserving decoding efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
