arXiv 2509.209122026-05-21cs.AI

DeFacto: 通过图像进行反事实推理以强制证据支持和忠实推理

DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning

Department of Automation, Tsinghua University, Beijing, China（清华大学自动化系）；Zhongguancun Academy, Beijing, China（中关村学院）；School of Software, Xinjiang University, Urumqi, China（新疆大学软件学院）；College of Materials Science and Engineering, Fuzhou University, Fuzhou, China（福州大学材料科学与工程学院）；Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院）；Beijing Qianjue Technology Co., Ltd., Beijing, China（北京千 jue 技术有限公司）

Tianrun Xu, Haoda Jing, Ye Li, Yuquan Wei, Jun Feng, Guanyu Chen, Haichuan Gao, Tianren Zhang, Feng Chen

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出DeFacto框架，通过整合正例、反事实和随机遮蔽三种训练范式，提升多模态语言模型在证据一致性方面的表现，并引入DeFacto-1.5K基准进行系统评估。

摘要：

最近多模态语言模型（MLLMs）的进步使通过图像进行推理成为多模态推理的主要范式。然而，现有方法仍无法确保答案与证据的一致性，即正确答案必须由正确视觉证据支持。为了解决这个问题，我们提出了DeFacto，一种反事实推理框架，该框架明确地将视觉证据与最终答案对齐。我们的方法整合了三种互补的训练范式：正例、反事实和随机遮蔽。我们进一步开发了一个语言引导的证据构建流水线，该流水线能够自动定位与问题相关区域并生成反事实变体，从而得到DeFacto-100K。基于此数据集，我们训练MLLMs使用基于GRPO的强化学习，并设计三种互补的奖励机制以促进正确回答、结构化推理和一致的证据选择。此外，我们引入了DeFacto-1.5K，一个由人类标注的基准，用于系统评估证据支持的一致性，而不仅仅是答案准确性。在多样化的基准测试中，DeFacto在答案准确性和证据-答案一致性方面均显著优于强大的基线模型。

英文摘要：

Recent advances in multimodal language models (MLLMs) have made thinking with images a dominant paradigm for multimodal reasoning. However, existing methods still fail to ensure evidence-answer consistency, where correct answers must be supported by correct visual evidence. To address this issue, we propose DeFacto, a counterfactual reasoning framework that explicitly aligns visual evidence with final answers. Our approach integrates three complementary training paradigms: positive, counterfactual, and random-masking. We further develop a language-guided evidence construction pipeline that automatically localizes question-relevant regions and generates counterfactual variants, resulting in DeFacto-100K. Building on this dataset, we train MLLMs with GRPO-based reinforcement learning and design three complementary rewards to promote correct answering, structured reasoning, and consistent evidence selection. Moreover, we introduce DeFacto-1.5K, a human-annotated benchmark for systematically evaluating evidence-grounded consistency beyond answer accuracy. Experiments on diverse benchmarks demonstrate that DeFacto substantially improves both answer accuracy and evidence-answer consistency over strong baselines.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。