arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01541cs.CVcs.AI
收藏

迈向多模态大语言模型的认知超感知

Toward Cognitive Supersensing in Multimodal Large Language Model

University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Boyi Li, Yifan Shen, Yuanzhe Liu, Yifan Xu, Jiateng Liu, Xinzhuo Li, Zhengyuan Li, Jingyuan Zhu, Yunhan Zhong, Fangzhou Lan, Jianguo Cao, James M. Rehg, Heng Ji… 展开作者

Boyi Li, Yifan Shen, Yuanzhe Liu, Yifan Xu, Jiateng Liu, Xinzhuo Li, Zhengyuan Li, Jingyuan Zhu, Yunhan Zhong, Fangzhou Lan, Jianguo Cao, James M. Rehg, Heng Ji, Ismini Lourentzou, Xu Cao

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出认知超感知方法，通过整合视觉图像预测头和强化学习阶段，提升多模态大语言模型在复杂认知任务中的表现，展现其在视觉问答基准中的优越性能。

摘要：

多模态大语言模型（MLLMs）在开放词汇感知任务中取得了显著成功，但其解决复杂认知问题的能力仍然有限，尤其是在视觉细节抽象且需要视觉记忆的情况下。当前方法主要在文本空间中扩展链式推理（CoT）推理，即使语言本身不足以提供清晰和结构化的推理，也 largely 忽略了类似于人类视觉空间草图和视觉图像的视觉推理机制。为缓解这一不足，我们引入了认知超感知，一种新的训练范式，通过整合一个潜在视觉图像预测（LVIP）头，使MLLMs具备类人视觉图像能力，通过联合学习视觉认知潜在嵌入序列并将其对齐到答案，从而形成基于视觉的内部推理链。我们进一步引入了一个强化学习阶段，基于此扎根的视觉潜在信息优化文本推理路径。为了评估MLLMs的认知能力，我们提出了CogSense-Bench，一个全面的视觉问答（VQA）基准，评估五个认知维度。广泛实验表明，使用认知超感知训练的MLLMs在CogSense-Bench上显著优于最先进的基线，并在超出领域数学和科学VQA基准上表现出更优的泛化能力，表明内部视觉图像可能是弥合感知识别与认知理解之间差距的关键因素。我们将开源CogSense-Bench和我们的模型权重。

英文摘要：

Multimodal Large Language Models (MLLMs) have achieved remarkable success in open-vocabulary perceptual tasks, yet their ability to solve complex cognitive problems remains limited, especially when visual details are abstract and require visual memory. Current approaches primarily scale Chain-of-Thought (CoT) reasoning in the text space, even when language alone is insufficient for clear and structured reasoning, and largely neglect visual reasoning mechanisms analogous to the human visuospatial sketchpad and visual imagery. To mitigate this deficiency, we introduce Cognitive Supersensing, a novel training paradigm that endows MLLMs with human-like visual imagery capabilities by integrating a Latent Visual Imagery Prediction (LVIP) head that jointly learns sequences of visual cognitive latent embeddings and aligns them with the answer, thereby forming vision-based internal reasoning chains. We further introduce a reinforcement learning stage that optimizes text reasoning paths based on this grounded visual latent. To evaluate the cognitive capabilities of MLLMs, we present CogSense-Bench, a comprehensive visual question answering (VQA) benchmark assessing five cognitive dimensions. Extensive experiments demonstrate that MLLMs trained with Cognitive Supersensing significantly outperform state-of-the-art baselines on CogSense-Bench and exhibit superior generalization on out-of-domain mathematics and science VQA benchmarks, suggesting that internal visual imagery is potentially key to bridging the gap between perceptual recognition and cognitive understanding. We will open-source the CogSense-Bench and our model weights.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑