arXiv 2601.164492026-02-23cs.CVcs.AI

Emotion-LLaMAv2 和 MMEVerse：多模态情感理解的新框架和基准

Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding

Shenzhen Technology University（深圳技术大学）；Shenzhen University of Information Technology（深圳信息科技学院）；University of Washington（华盛顿大学）；Foundation Model Team, Meituan（美团基础模型团队）；National University of Singapore（新加坡国立大学）；Tongji University（同济大学）

Xiaojiang Peng, Jingyi Chen, Zebang Cheng, Bao Peng, Fengyi Wu, Yifei Dong, Shuyuan Tu, Qiyu Hu, Huiting Huang, Yuxiang Lin, Jun-Yan He, Kai Wang, Zheng Lian, Zhi-Qi Cheng

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

Emotion-LLaMAv2和MMEVerse通过端到端多视图编码器、Conv Attention预融合模块和课程指令调制方案，提升了多模态情绪理解和推理能力，并构建了统一的多模态情绪基准。

摘要：

从多模态信号中理解人类情绪在情感计算和人机交互领域构成了重大挑战。尽管多模态大语言模型（MLLMs）在一般视觉-语言任务中表现出色，但其在情感推理方面的能力仍然有限。目前领域面临大规模高质量描述性情绪标注数据集匮乏以及缺乏标准化评估基准的问题。我们的初步框架Emotion-LLaMA开创了用于情感推理的指令调制多模态学习，但受限于显式面部检测器、隐式融合策略和低质量训练数据规模有限。为解决这些限制，我们提出了Emotion-LLaMAv2和MMEVerse基准，建立了一个端到端的管道以及标准化的情感识别和推理评估设置。Emotion-LLaMAv2引入了三个关键改进。首先，一个端到端的多视图编码器消除了外部面部检测，并通过更丰富的空间和时间多视图标记捕捉细微的情绪线索。其次，设计了一个Conv Attention预融合模块，以在LLM骨干外部同时实现局部和全局多模态特征交互。第三，一个感知到认知的课程指令调制方案在LLaMA2骨干内统一了情绪识别和自由形式的情绪推理。为了支持大规模训练和可重复的评估，MMEVerse汇集了十二个公开可用的情绪数据集，包括IEMOCAP、MELD、DFEW和MAFW，转化为统一的多模态指令格式。数据通过一个涉及Qwen2 Audio、Qwen2.5 VL和GPT 4o的多代理流程重新标注，产生130k训练片段和36k测试片段，覆盖18个评估基准。

英文摘要：

Understanding human emotions from multimodal signals poses a significant challenge in affective computing and human-robot interaction. While multimodal large language models (MLLMs) have excelled in general vision-language tasks, their capabilities in emotional reasoning remain limited. The field currently suffers from a scarcity of large-scale datasets with high-quality, descriptive emotion annotations and lacks standardized benchmarks for evaluation. Our preliminary framework, Emotion-LLaMA, pioneered instruction-tuned multimodal learning for emotion reasoning but was restricted by explicit face detectors, implicit fusion strategies, and low-quality training data with limited scale. To address these limitations, we present Emotion-LLaMAv2 and the MMEVerse benchmark, establishing an end-to-end pipeline together with a standardized evaluation setting for emotion recognition and reasoning. Emotion-LLaMAv2 introduces three key advances. First, an end-to-end multiview encoder eliminates external face detection and captures nuanced emotional cues via richer spatial and temporal multiview tokens. Second, a Conv Attention pre-fusion module is designed to enable simultaneous local and global multimodal feature interactions external to the LLM backbone. Third, a perception-to-cognition curriculum instruction tuning scheme within the LLaMA2 backbone unifies emotion recognition and free-form emotion reasoning. To support large-scale training and reproducible evaluation, MMEVerse aggregates twelve publicly available emotion datasets, including IEMOCAP, MELD, DFEW, and MAFW, into a unified multimodal instruction format. The data are re-annotated via a multi-agent pipeline involving Qwen2 Audio, Qwen2.5 VL, and GPT 4o, producing 130k training clips and 36k testing clips across 18 evaluation benchmarks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。