ReLoop：通过闭环训练实现“两次观察与逆向思考”，以缓解多模态理解中的幻觉
ReLoop: "Seeing Twice and Thinking Backwards" via Closed-loop Training to Mitigate Hallucinations in Multimodal understanding
AI总结：
本文提出 ReLoop 闭环训练框架，通过冻结的 CFP 融合语义重建、视觉描述和注意力监督，使 MLLMs 在训练中自我校验并降低多模态幻觉。
摘要：
尽管多模态大语言模型（MLLMs）在开放式视觉问答中取得了显著进展，但它们仍易受幻觉影响。幻觉是指与输入语义相矛盾或错误表征输入语义的输出，对可靠性和事实一致性构成关键挑战。现有方法通常依赖外部验证或事后纠正，缺乏在训练期间直接验证输出的内部机制。为弥补这一差距，我们提出 ReLoop，这是一个统一的闭环训练框架，旨在促进 MLLMs 跨模态理解中的多模态一致性。ReLoop 采用环形结构，集成三种互补的一致性反馈机制，迫使 MLLMs“两次观察并逆向思考”。具体而言，ReLoop 使用冻结的 Consistency Feedback Plugin（CFP，一致性反馈插件），该插件由语义重建、视觉描述以及用于注意力对齐的注意力监督模块组成。这些组件共同强化语义可逆性、视觉一致性和可解释的注意力，使模型能够在训练期间纠正其输出。大量评估和分析证明，ReLoop 在多个基准上有效降低了幻觉率，为缓解 MLLMs 中的幻觉建立了一种稳健方法。我们将在 camera-ready 版本中发布源代码和数据。
英文摘要：
While Multimodal Large Language Models (MLLMs) have achieved remarkable progress in open-ended visual question answering, they remain vulnerable to hallucinations. These are outputs that contradict or misrepresent input semantics, posing a critical challenge to the reliability and factual consistency. Existing methods often rely on external verification or post-hoc correction, lacking an internal mechanism to validate outputs directly during training. To bridge this gap, we propose ReLoop, a unified closed-loop training framework that encourages multimodal consistency for cross-modal understanding in MLLMs. ReLoop adopts a ring-shaped structure that integrates three complementary consistency feedback mechanisms, obliging MLLMs to "seeing twice and thinking backwards". Specifically, ReLoop employs the frozen Consistency Feedback Plugin (CFP), comprising semantic reconstruction, visual description, and an attention supervision module for attention alignment. These components collectively enforce semantic reversibility, visual consistency, and interpretable attention, enabling the model to correct its outputs during training. Extensive evaluations and analyses demonstrate the effectiveness of ReLoop in reducing hallucination rates across multiple benchmarks, establishing a robust method for hallucination mitigation in MLLMs. We will release our source code and data in the camera-ready version.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
