利用指令对比解码缓解大型视觉语言模型中的幻觉
Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
AI总结：
本文提出指令对比解码（ICD）方法，通过对比标准和指令扰动分布来减少大型视觉语言模型推理时的幻觉，显著缓解对象级和属性级幻觉并增强模型的通用感知与识别能力。
摘要：
大型视觉语言模型（LVLMs）越来越擅长从视觉输入中生成上下文详细且连贯的响应。然而，它们在多模态决策和开放式生成中的应用受到了显著幻觉率的阻碍，生成的文本不准确地表征了视觉内容。为了解决这个问题，本文引入了指令对比解码（ICD）方法，这是一种旨在减少LVLM推理期间幻觉的新颖方法。我们的方法受到我们观察的启发，即我们所谓的扰动指令显著加剧了多模态融合模块中的幻觉。ICD对比标准和指令扰动的分布，从而增加对齐不确定性，并有效地从原始分布中减去幻觉概念。通过在判别性基准（POPE和MME）和生成性基准（LLaVa-Bench）上的全面实验，我们证明ICD显著缓解了对象级和属性级幻觉。此外，我们的方法不仅解决了幻觉问题，还显著增强了LVLMs的通用感知和识别能力。
英文摘要：
Large Vision-Language Models (LVLMs) are increasingly adept at generating contextually detailed and coherent responses from visual inputs. However, their application in multimodal decision-making and open-ended generation is hindered by a notable rate of hallucinations, where generated text inaccurately represents the visual contents. To address this issue, this paper introduces the Instruction Contrastive Decoding (ICD) method, a novel approach designed to reduce hallucinations during LVLM inference. Our method is inspired by our observation that what we call disturbance instructions significantly exacerbate hallucinations in multimodal fusion modules. ICD contrasts distributions from standard and instruction disturbance, thereby increasing alignment uncertainty and effectively subtracting hallucinated concepts from the original distribution. Through comprehensive experiments on discriminative benchmarks (POPE and MME) and a generative benchmark (LLaVa-Bench), we demonstrate that ICD significantly mitigates both object-level and attribute-level hallucinations. Moreover, our method not only addresses hallucinations but also significantly enhances the general perception and recognition capabilities of LVLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
