PostAlign：多模态对齐作为MLLMs的校正透镜
PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs
AI总结：
本文提出MMGrounded-PostAlign框架，通过多模态接地和负拒绝机制增强MLLMs视觉理解并抑制幻觉，在多个基准上取得显著改进。
摘要：
多模态大语言模型（MLLMs）在视觉-语言任务中表现出色，如图像描述和视觉问答。然而，它们常常过度依赖虚假相关性，这主要是由于语言先验分散了模型对实际视觉信息的利用。为解决这些问题，我们引入了MMGrounded-PostAlign，一个后多模态对齐框架，旨在增强MLLMs的视觉理解能力并减轻其幻觉。我们的框架包含一个多模态接地模块，既用于视觉接地（识别图像中引用的对象），也用于文本接地（为最终答案生成理由），确保输出基于视觉和文本证据。为减轻幻觉，我们在视觉接地模块中引入了一种负拒绝机制，以区分被接地的实体与受语言偏见影响的不存在的对象。在文本接地方面，我们提出了一种选择性推理机制，根据查询复杂性调整模型的推理策略。我们在POPE、HaloQuest、VQAv2、MME和MMBench等基准上进行了广泛评估，显示出在细粒度视觉理解和幻觉抑制方面的显著改进。
英文摘要：
Multimodal Large Language Models (MLLMs) excel in vision-language tasks, such as image captioning and visual question answering. However, they often suffer from over-reliance on spurious correlations, primarily due to linguistic priors that distract the model from leveraging actual visual information. To address these issues, we introduce MMGrounded-PostAlign, a post-multimodal alignment framework designed to enhance the visual understanding capabilities and mitigate the hallucinations of MLLMs. Our framework incorporates a multimodal grounding module for both visual grounding, which identifies the referred object in the image, and textual grounding, which generates the rationale for the final answer, ensuring that outputs are anchored in both visual and textual evidence. To mitigate the hallucinations, we introduce a negative rejection mechanism in the visual grounding module to distinguish grounded entities from non-existent objects influenced by linguistic biases. On the textual grounding side, we propose a selective reasoning mechanism that adjusts the model's reasoning strategy based on query complexity. Extensive evaluations are conducted on benchmarks such as POPE, HaloQuest, VQAv2, MME, and MMBench showing significant improvements in fine-grained visual understanding and hallucination suppression.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
