通过视觉对比解码缓解大型视觉语言模型中的物体幻觉
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
AI总结：
本文提出无需训练的视觉对比解码方法 VCD，通过对比原始与失真视觉输入的输出分布，减少统计偏差和单模态先验依赖，从而显著缓解 LVLMs 的物体幻觉问题，并在通用基准上表现优异。
摘要：
大型视觉语言模型（LVLMs）已取得显著进展，将视觉识别与语言理解交织在一起，生成不仅连贯而且贴合上下文的内容。尽管取得了成功，LVLMs 仍然面临物体幻觉问题，即模型生成看似合理但不正确的输出，其中包含图像中并不存在的物体。为了缓解这一问题，我们提出了视觉对比解码（Visual Contrastive Decoding，VCD），这是一种简单且无需训练的方法，通过对比原始视觉输入和失真视觉输入所得到的输出分布来工作。所提出的 VCD 有效减少了对统计偏差和单模态先验的过度依赖，而这两者是物体幻觉的两个重要成因。这种调整确保生成的内容紧密基于视觉输入，从而产生上下文准确的输出。我们的实验表明，VCD 无需额外训练或使用外部工具，即可在不同 LVLM 家族中显著缓解物体幻觉问题。除了缓解物体幻觉之外，VCD 在通用 LVLM 基准测试中也表现出色，凸显了其广泛的适用性。
英文摘要：
Large Vision-Language Models (LVLMs) have advanced considerably, intertwining visual recognition and language understanding to generate content that is not only coherent but also contextually attuned. Despite their success, LVLMs still suffer from the issue of object hallucinations, where models generate plausible yet incorrect outputs that include objects that do not exist in the images. To mitigate this issue, we introduce Visual Contrastive Decoding (VCD), a simple and training-free method that contrasts output distributions derived from original and distorted visual inputs. The proposed VCD effectively reduces the over-reliance on statistical bias and unimodal priors, two essential causes of object hallucinations. This adjustment ensures the generated content is closely grounded to visual inputs, resulting in contextually accurate outputs. Our experiments show that VCD, without either additional training or the usage of external tools, significantly mitigates the object hallucination issue across different LVLM families. Beyond mitigating object hallucinations, VCD also excels in general LVLM benchmarks, highlighting its wide-ranging applicability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
