通过自下而上的整体推理来对抗多模态LLM幻觉
Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning
AI总结：
本文提出一种自下而上的整体推理框架，通过验证并整合感知层面信息与认知层面常识知识，系统解决多模态大语言模型中视觉和文本输入的潜在问题，有效对抗幻觉并显著提升多个幻觉基准上的表现。
摘要：
多模态大语言模型（MLLMs）的最新进展在推进各种视觉-语言任务方面展现出前所未有的能力。然而，MLLMs面临幻觉的重大挑战，即产生与输入数据不一致的误导性输出。尽管已有努力致力于对抗MLLM幻觉，但几个关键挑战仍未解决。首先，当前方法激进地聚焦于解决感知层面的错误，而另一种需要事实常识的认知层面的重要错误类型却可能被忽视。此外，现有方法在寻找更有效的视觉输入表示方式上可能存在不足，而这正是引发视觉幻觉的关键瓶颈。此外，MLLMs经常被错误的文本输入误导并产生幻觉，而不幸的是，这类问题长期被现有研究忽视。受人类处理幻觉直觉的启发，本文引入了一种新颖的自下而上推理框架。我们的框架通过验证并整合感知层面的信息与认知层面的常识知识，系统地解决了视觉和文本输入中的潜在问题，确保更可靠的输出。大量实验表明，将MLLMs与所提框架整合后，在多个幻觉基准上取得了显著改进。深入分析揭示了我们的方法在解决感知层面和认知层面幻觉方面的巨大潜力。
英文摘要：
Recent advancements in multimodal large language models (MLLMs) have shown unprecedented capabilities in advancing various vision-language tasks. However, MLLMs face significant challenges with hallucinations, and misleading outputs that do not align with the input data. While existing efforts are paid to combat MLLM hallucinations, several pivotal challenges are still unsolved. First, while current approaches aggressively focus on addressing errors at the perception level, another important type at the cognition level requiring factual commonsense can be overlooked. In addition, existing methods might fall short in finding a more effective way to represent visual input, which is yet a key bottleneck that triggers visual hallucinations. Moreover, MLLMs can frequently be misled by faulty textual inputs and cause hallucinations, while unfortunately, this type of issue has long been overlooked by existing studies. Inspired by human intuition in handling hallucinations, this paper introduces a novel bottom-up reasoning framework. Our framework systematically addresses potential issues in both visual and textual inputs by verifying and integrating perception-level information with cognition-level commonsense knowledge, ensuring more reliable outputs. Extensive experiments demonstrate significant improvements in multiple hallucination benchmarks after integrating MLLMs with the proposed framework. In-depth analyses reveal the great potential of our methods in addressing perception- and cognition-level hallucinations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
