利用层级对比解码缓解多模态大语言模型中的幻觉
Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding
AI总结：
针对多模态大语言模型的图像幻觉问题，提出 LayerCD，通过对比视觉编码器浅层与深层特征产生的输出分布过滤幻觉，并在两个基准上显著超过现有方法。
摘要：
多模态大语言模型（MLLM）已展现出令人瞩目的感知与推理能力，却经常受到幻觉困扰——即生成语言上连贯、但与输入图像上下文不一致的输出，包括物体、属性和关系方面的错误。为应对这一挑战，我们提出了一种名为层级对比解码（Layer Contrastive Decoding，LayerCD）的简单方法。我们的设计源于一项观察：浅层视觉特征比深层视觉特征更有可能导致 MLLM 产生幻觉，因为它们仅捕捉有偏的低层信息，不足以支持高层推理。因此，LayerCD 旨在通过对比由不同层级视觉特征生成的输出分布来过滤幻觉，具体而言，分别对比来自视觉编码器浅层和深层的特征。我们在两个幻觉基准上进行了大量实验，结果表明 LayerCD 显著优于当前最先进方法。LayerCD 的代码可在 https://github.com/maifoundations/LayerCD 获取。
英文摘要：
Multimodal Large Language Models (MLLMs) have shown impressive perception and reasoning capabilities, yet they often suffer from hallucinations -- generating outputs that are linguistically coherent but inconsistent with the context of the input image, including inaccuracies in objects, attributes, and relations. To address this challenge, we propose a simple approach called Layer Contrastive Decoding (LayerCD). Our design is motivated by the observation that shallow visual features are much more likely than deep visual features to cause an MLLM to hallucinate as they only capture biased, low-level information that is insufficient for high-level reasoning. Therefore, LayerCD aims to filter out hallucinations by contrasting the output distributions generated from visual features of different levels, specifically those from the shallow and deep layers of the vision encoder, respectively. We conduct extensive experiments on two hallucination benchmarks and show that LayerCD significantly outperforms current state-of-the-art. The code for LayerCD is available at https://github.com/maifoundations/LayerCD .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
