Mixture of Decoding：一种受注意力启发的自适应解码策略，用于缓解大型视觉-语言模型中的幻觉
Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
AI总结：
本文提出 Mixture of Decoding（MoD），通过比较原始与被注意图像 token 的输出一致性，自适应采用互补或对比解码策略，从而在多个基准上有效缓解 LVLMs 幻觉。
摘要：
大型视觉-语言模型（LVLMs）已在各类视觉任务中展现出令人瞩目的能力，但仍受到幻觉这一持续挑战的阻碍。为解决这一关键问题，我们提出 Mixture of Decoding（MoD），这是一种用于缓解幻觉的新颖方法，通过评估模型对图像 token 的注意力正确性来动态调整解码策略。具体而言，MoD 测量由原始图像 token 生成的输出与由模型所注意的图像 token 得出的输出之间的一致性，以辨别上述正确性。如果输出一致，表明注意力正确，MoD 采用互补策略来放大关键信息。相反，如果输出不一致，表明注意力错误，MoD 利用对比策略来抑制误导性信息。大量实验表明，MoD 在多个主流基准上显著优于现有解码方法，有效缓解了 LVLMs 中的幻觉。代码可在 https://github.com/xlchen0205/MoD 获取。
英文摘要：
Large Vision-Language Models (LVLMs) have exhibited impressive capabilities across various visual tasks, yet they remain hindered by the persistent challenge of hallucinations. To address this critical issue, we propose Mixture of Decoding (MoD), a novel approach for hallucination mitigation that dynamically adapts decoding strategies by evaluating the correctness of the model's attention on image tokens. Specifically, MoD measures the consistency between outputs generated from the original image tokens and those derived from the model's attended image tokens, to distinguish the correctness aforementioned. If the outputs are consistent, indicating correct attention, MoD employs a complementary strategy to amplify critical information. Conversely, if the outputs are inconsistent, suggesting erroneous attention, MoD utilizes a contrastive strategy to suppress misleading information. Extensive experiments demonstrate that MoD significantly outperforms existing decoding methods across multiple mainstream benchmarks, effectively mitigating hallucinations in LVLMs. The code is available at https://github.com/xlchen0205/MoD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
