arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.07388cs.CL
收藏

多模态情感计算近期进展:自然语言处理视角

Recent Advances in Multimodal Affective Computing: An NLP Perspective

Guangdong University of Technology(广东工业大学);Stony Brook University(石溪大学);University of Bologna(博洛尼亚大学);Peking University(北京大学);Nanyang Technological University(南洋理工大学);Arizona State University(亚利桑那州立大学)

Guimin Hu, Weimin Lyu, Chang Sun, Zhihong Zhu, Lin Gui, Ruichu Cai, Erik Cambria, Hasti Seifi

AI总结:

本文从自然语言处理视角系统回顾了多模态情感计算的最新进展,聚焦于多模态情感分析、对话中多模态情绪识别、基于方面的多模态情感分析及多标签情绪识别等四个任务,提出统一视角并探讨相关方向与挑战。

摘要:

多模态情感计算因其在理解人类行为和意图方面的广泛应用而日益受到关注,特别是在以文本为中心的多模态场景中。现有研究涵盖多样化的任务、模态和建模范式,但缺乏统一的视角。本文从自然语言处理视角系统回顾了近期进展,重点讨论四个代表性任务:多模态情感分析(MSA)、对话中多模态情绪识别(MERC)、基于方面的多模态情感分析(MABSA)和多模态多标签情绪识别(MMER)。通过比较任务定义、基准数据集和评估协议,并将代表性方法组织到关键范式中,包括多任务学习、预训练模型、知识增强和上下文建模。进一步探讨了面部、声学和生理模态及相关方向,如情绪原因分析。最后,本文强调了关键挑战并概述了有前景的未来方向。为促进进一步研究,我们发布了一个经过精心整理的相关工作和资源库。\footnote{https://anonymous.4open.science/r/Multimodal-Affective-Computing-Survey-9819}

英文摘要:

Multimodal affective computing has gained increasing attention due to its broad applications in understanding human behavior and intentions, particularly in text-centric multimodal scenarios. Existing research spans diverse tasks, modalities, and modeling paradigms, yet lacks a unified perspective. In this survey, we systematically review recent advances from an NLP perspective, focusing on four representative tasks: multimodal sentiment analysis (MSA), multimodal emotion recognition in conversation (MERC), multimodal aspect-based sentiment analysis (MABSA), and multimodal multi-label emotion recognition (MMER). We present a unified view by comparing task formulations, benchmark datasets, and evaluation protocols, and by organizing representative methods into key paradigms, including multitask learning, pre-trained models, knowledge enhancement, and contextual modeling. We further extend the discussion to related directions, such as facial, acoustic, and physiological modalities, as well as emotion cause analysis. Finally, we highlight key challenges and outline promising future directions. To facilitate further research, we release a curated repository of relevant works and resources \footnote{https://anonymous.4open.science/r/Multimodal-Affective-Computing-Survey-9819}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑