MaskCD：通过图像头掩码对比解码缓解LVLM幻觉
MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
AI总结：
针对大视觉语言模型的幻觉问题，提出利用图像头掩码构建对比样本的MaskCD方法，在多个模型和基准上验证其可缓解幻觉且保留模型通用能力。
摘要：
大视觉语言模型（LVLM）在下游多模态任务的视觉-语言理解中表现出卓越性能。随着其能力不断提升，问题也随之出现。其中幻觉问题备受关注，指的是LVLM生成与输入视觉和文本内容相矛盾内容的现象。已有诸多方法被提出以解决该问题，例如对比解码和注意力操控。然而，对比解码方法难以构建合适的对比样本，注意力操控方法则敏感度高，缺乏稳定性。在本研究中，我们提出图像头掩码对比解码（MaskCD）。该方法利用LVLM中的“图像头”，通过掩码处理构建对比样本以进行对比解码。我们在LLaVA-1.5-7b和Qwen-VL-7b上对MaskCD进行了评估，使用了CHAIR、POPE、AMBER和MME等多种基准。结果表明，MaskCD有效缓解了幻觉现象，同时保留了LVLM的通用能力。相关资源可在以下网址获取：https://github.com/Deng-Jingyuan/MaskCD 。
英文摘要：
Large vision-language models (LVLMs) have shown remarkable performance in visual-language understanding for downstream multimodal tasks. While their capabilities are improving, problems emerge simultaneously. Among those problems, the hallucinations have attracted much attention, which stands for the phenomenon where LVLMs generate contradictory content to their input visual and text contents. Many approaches have been proposed to deal with this issue, such as contrastive decoding and attention manipulation. However, contrastive decoding methods struggle in constructing appropriate contrastive samples, and attention manipulation methods are highly sensitive, lacking stability. In this work, we propose image head Masked Contrastive Decoding (MaskCD). Our approach utilizes the "image heads" in LVLMs, masking them to construct contrastive samples for contrastive decoding. We evaluated MaskCD on LLaVA-1.5-7b and Qwen-VL-7b, using various benchmarks such as CHAIR, POPE, AMBER and MME. The results demonstrate that MaskCD effectively alleviates the phenomenon of hallucinations and retains the general capabilities of LVLMs. Corresponding resources could be found at: https://github.com/Deng-Jingyuan/MaskCD .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
