arXiv 2411.077222025-11-12cs.AI

认知与感知是否一致？评估并缓解文档理解中的多模态知识冲突

Is Cognition Consistent with Perception? Assessing and Mitigating Multimodal Knowledge Conflicts in Document Understanding

Zhejiang Key Laboratory of Accessible Perception and Intelligent Systems, Zhejiang University（浙江可感知智能系统重点实验室，浙江大学）；Alibaba Group（阿里巴巴集团）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and DataSecurity（杭州高新技术区（滨江）区块链与数据安全研究院）

Zirui Shao, Feiyu Gao, Zhaoqing Zhu, Chuwei Luo, Hangdi Xing, Zhi Yu, Qi Zheng, Ming Yan, Jiajun Bu

备注：: Accepted at EMNLP 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对文档理解中MLLMs面临的感知与认知冲突问题，本文定义并评估了C&P知识冲突，提出多模态知识一致性微调方法，有效降低冲突并提升模型在两类任务中的性能。

摘要：

多模态大语言模型（MLLMs）在文档理解领域展现出令人印象深刻的能力，该领域是一个快速发展且具有显著工业需求的研究方向。作为一项多模态任务，文档理解要求模型同时具备感知和认知能力。然而，由于训练中存在不同类型的标注噪声，当前MLLMs常面临感知与认知之间的冲突。以文档VQA任务（认知）为例，MLLM生成的答案可能与其OCR（感知）识别的对应视觉内容不匹配。这种冲突表明MLLM可能难以在“所见”信息与“所理解”信息间建立内在联系，挑战了认知与感知一致的直觉观念，阻碍了MLLMs的性能与可解释性。本文将认知与感知间的冲突定义为认知感知（C&P）知识冲突（一种多模态知识冲突），并针对文档理解领域系统评估该冲突。分析显示，即使是领先的MLLM GPT-4o，其C&P一致性也仅达75.26%。为缓解C&P知识冲突，我们提出一种名为多模态知识一致性微调的新方法。该方法降低了所有测试MLLMs的C&P知识冲突，并提升了它们在认知和感知任务中的性能。

英文摘要：

Multimodal large language models (MLLMs) have shown impressive capabilities in document understanding, a rapidly growing research area with significant industrial demand. As a multimodal task, document understanding requires models to possess both perceptual and cognitive abilities. However, due to different types of annotation noise in training, current MLLMs often face conflicts between perception and cognition. Taking a document VQA task (cognition) as an example, an MLLM might generate answers that do not match the corresponding visual content identified by its OCR (perception). This conflict suggests that the MLLM might struggle to establish an intrinsic connection between the information it "sees" and what it "understands". Such conflicts challenge the intuitive notion that cognition is consistent with perception, hindering the performance and explainability of MLLMs. In this paper, we define the conflicts between cognition and perception as Cognition and Perception (C&P) knowledge conflicts, a form of multimodal knowledge conflict, and systematically assess them with a focus on document understanding. Our analysis reveals that even GPT-4o, a leading MLLM, achieves only 75.26% C&P consistency. To mitigate the C&P knowledge conflicts, we propose a novel method called Multimodal Knowledge Consistency Fine-tuning. Our method reduces C&P knowledge conflicts across all tested MLLMs and enhances their performance in both cognitive and perceptual tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。