arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.04073cs.CVcs.AIcs.CL
收藏

在跨模态冲突下分析大型多模态模型的推理一致性

Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts

Harbin Institute of Technology(哈尔滨工业大学);Peng Cheng Laboratory(鹏城实验室);National University of Singapore(新加坡国立大学)

Zhihao Zhu, Jiafeng Liang, Shixin Jiang, Jinlan Fu, Ming Liu, Guanglu Sun, See-Kiong Ng, Bing Qin

AI总结:

本文提出逻辑图扰动协议,通过主动视觉上下文细化方法,解决大型多模态模型在跨模态冲突下的推理一致性问题,提升推理鲁棒性。

摘要:

大型多模态模型(LMMs)在通过链式推理(CoT)进行视频推理方面展现了出色的性能。然而,其推理链的鲁棒性仍存疑。在本文中,我们识别出一种关键的失败模式,称为文本惯性,即一旦推理过程中出现文本幻觉,模型倾向于盲目遵循错误文本,而忽视冲突的视觉证据。为了系统研究这一问题,我们提出了逻辑图扰动协议,该协议在不同LMMs的推理链中结构性地注入扰动,涵盖原生推理架构和提示驱动范式,以评估其自我反思能力。结果表明,模型在少于10%的情况下能够成功自我纠正,而大部分模型则陷入盲目的文本错误传播。为缓解这一问题,我们引入了主动视觉上下文细化,一种无需训练的推理范式,它通过主动视觉重新定位机制来强制细粒度验证,并结合自适应上下文细化策略来总结和去噪推理历史。实验表明,我们的方法显著抑制了幻觉传播并增强了推理的鲁棒性。

英文摘要:

Large Multimodal Models (LMMs) have demonstrated impressive capabilities in video reasoning via Chain-of-Thought (CoT). However, the robustness of their reasoning chains remains questionable. In this paper, we identify a critical failure mode termed textual inertia, where once a textual hallucination occurs in the thinking process, models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence. To systematically investigate this, we propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the reasoning chains of diverse LMMs spanning both native reasoning architectures and prompt-driven paradigms to evaluate their self-reflection capabilities. The results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation. To mitigate this, we introduce Active Visual-Context Refinement, a training-free inference paradigm which orchestrates an active visual re-grounding mechanism to enforce fine-grained verification coupled with an adaptive context refinement strategy to summarize and denoise the reasoning history. Experiments demonstrate that our approach significantly stifles hallucination propagation and enhances reasoning robustness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑