arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00971cs.CV
收藏

揭示认知罗盘:基于理论of-Mind的多模态情感推理

Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoning

National University of Singapore(新加坡国立大学);Huazhong University of Science and Technology(华中科技大学);The Hong Kong Polytechnic University(香港理工大学)

Meng Luo, Bobo Li, Shanqing Xu, Shize Zhang, Qiuchan Chen, Menglu Han, Wenhao Chen, Yanxiang Huang, Hao Fei, Mong-Li Lee, Wynne Hsu

AI总结:

本文提出HitEmotion基准和TMPO方法,通过理论of-Mind引导多模态情感推理,提升模型情感理解和认知能力。

摘要:

尽管多模态大语言模型(MLLMs)在多模态领域取得了快速进展,但其在深度情感理解方面的能力仍然有限。我们主张,真正的情感智能需要显式建模理论of-Mind(ToM),这是情感产生的认知基础。为此,我们引入了HitEmotion,一个基于ToM的分层基准,用于诊断模型在不同认知深度层次上的能力断点。其次,我们提出了一种基于ToM的推理链,用于跟踪心理状态并校准跨模态证据以实现忠实的情感推理。我们进一步引入TMPO,一种强化学习方法,利用中间心理状态作为过程级监督来引导和加强模型推理。广泛的实验表明,HitEmotion揭示了最先进的模型在深度情感推理方面的缺陷,尤其是在认知 demanding 任务上。在评估中,基于ToM的推理链和TMPO提高了最终任务的准确性,并产生了更加忠实和连贯的推理过程。最后,我们的工作为研究社区提供了一个实用的工具包,用于评估和增强MLLMs基于认知的情感理解能力。我们的数据集和代码可在:https://HitEmotion.github.io/上找到。

英文摘要:

Despite rapid progress in multimodal large language models (MLLMs), their capability for deep emotional understanding remains limited. We argue that genuine affective intelligence requires explicit modeling of Theory of Mind (ToM), the cognitive substrate from which emotions arise. To this end, we introduce HitEmotion, a ToM-grounded hierarchical benchmark that diagnoses capability breakpoints across increasing levels of cognitive depth. Second, we propose a ToM-guided reasoning chain that tracks mental states and calibrates cross-modal evidence to achieve faithful emotional reasoning. We further introduce TMPO, a reinforcement learning method that uses intermediate mental states as process-level supervision to guide and strengthen model reasoning. Extensive experiments show that HitEmotion exposes deep emotional reasoning deficits in state-of-the-art models, especially on cognitively demanding tasks. In evaluation, the ToM-guided reasoning chain and TMPO improve end-task accuracy and yield more faithful, more coherent rationales. In conclusion, our work provides the research community with a practical toolkit for evaluating and enhancing the cognition-based emotional understanding capabilities of MLLMs. Our dataset and code are available at: https://HitEmotion.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑