Recent Advances in Multimodal Affective Computing: An NLP Perspective
Guangdong University of Technology(广东工业大学);Stony Brook University(石溪大学);University of Bologna(博洛尼亚大学);Peking University(北京大学);Nanyang Technological University(南洋理工大学);Arizona State University(亚利桑那州立大学)
Guimin Hu, Weimin Lyu, Chang Sun, Zhihong Zhu, Lin Gui, Ruichu Cai, Erik Cambria, Hasti Seifi
Multimodal affective computing has gained increasing attention due to its broad applications in understanding human behavior and intentions, particularly in text-centric multimodal scenarios. Existing research spans diverse tasks, modalities, and modeling paradigms, yet lacks a unified perspective. In this survey, we systematically review recent advances from an NLP perspective, focusing on four representative tasks: multimodal sentiment analysis (MSA), multimodal emotion recognition in conversation (MERC), multimodal aspect-based sentiment analysis (MABSA), and multimodal multi-label emotion recognition (MMER). We present a unified view by comparing task formulations, benchmark datasets, and evaluation protocols, and by organizing representative methods into key paradigms, including multitask learning, pre-trained models, knowledge enhancement, and contextual modeling. We further extend the discussion to related directions, such as facial, acoustic, and physiological modalities, as well as emotion cause analysis. Finally, we highlight key challenges and outline promising future directions. To facilitate further research, we release a curated repository of relevant works and resources \footnote{https://anonymous.4open.science/r/Multimodal-Affective-Computing-Survey-9819}.
Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysis
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Saw Swee Hock School of Public Health, National University of Singapore(新加坡国立大学 Saw Swee Hock 公共卫生学院)
Affective Multimodal Agents with Proactive Knowledge Grounding for Emotionally Aligned Marketing Dialogue
Hunan Police Academy, Department of Criminal Investigation(湖南警察学院犯罪侦查系);Business College, California State University(加州州立大学长滩分校商学院);Northwestern University(西北大学);AI Agent Lab, Vokram Group(Vokram集团AI代理实验室);Beijing University of Technology(北京理工大学);Zheng Zhou University of Light Industry(郑州轻工业大学)
Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models
Center for Language and Speech Processing (CLSP), Johns Hopkins University(约翰霍普金斯大学语言与语音处理中心);Group on Language, Audio & Music (GLAM), Imperial College London(伦敦帝国理工学院语言、音频与音乐组)
GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations
Imperial College London(伦敦帝国学院);Technical University of Munich(慕尼黑工业大学);Konrad Zuse School of Excellence in Reliable AI(康拉德·祖斯可靠人工智能卓越学院);Munich Data Science Institute(慕尼黑数据科学研究所);Munich Center for Machine Learning(慕尼黑机器学习中心)
Semantic Matters: Multimodal Features for Affective Analysis
University of Augsburg(奥格斯堡大学);University of the Bundeswehr Munich(慕尼黑联邦国防军大学);Chair for Human-Centered Artificial Intelligence(以人为中心的人工智能主席席位);Institute for Distributed Intelligent Systems(分布式智能系统研究所)
MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
National University of Singapore(新加坡国立大学);Instituto Politécnico Nacional(墨西哥国立理工学院);Nanyang Technological University(南洋理工大学);University of Michigan(密歇根大学)
Affective Computing Has Changed: The Foundation Model Disruption
Technical University of Munich(慕尼黑工业大学);Munich Data Science Institute(慕尼黑数据科学研究所);Munich Center for Machine Learning(慕尼黑机器学习中心);Imperial College London(伦敦帝国学院);University of Augsburg(奥格斯堡大学);Universidad Autonoma de Madrid(马德里自治大学);SyncPilot GmbH(SyncPilot有限公司)
Echoes of Unrest: A Multimodal NLP Framework for Early Warning of Fake News and Violence-Driven Mob Activity
Department of Computer Science and Engineering, Dhaka International University(达卡国际大学计算机科学与工程系);Department of Computer Science and Engineering, Bangladesh University of Engineering and Technology(孟加拉国工程技术大学计算机科学与工程系)
Language and Multimodal Models in Sports: A Survey of Datasets and Applications
University of California, Irvine(加利福尼亚大学欧文分校);George Mason University(乔治梅森大学);Meta Platforms, Inc.(元平台公司);Stanford University(斯坦福大学);University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);Rice University(莱斯大学)
When LLMs Team Up: The Emergence of Collaborative Affective Computing
Department of Computing, Hong Kong Polytechnic University(香港理工大学计算机系);School of Data Science, Lingnan University(岭南大学数据科学学院);School of Computer Science and the Data Science Institute, University of Technology Sydney(悉尼大学计算机科学学院和数据科学研究所);Education University of Hong Kong(香港教育大学)
MER-Bench: Benchmarking Multimodal Emotional Reappraisal for Image-Text Memes
School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机科学与信息工程学院);College of Information Technology, United Arab Emirates University(阿拉伯联合酋长国大学信息学院);Institute of Advanced Technology, University of Science and Technology of China(中国科学技术大学先进技术研究院)
A Chinese Multi-label Affective Computing Dataset Based on Social Media Network Users
Beijing Institute of Technology(北京理工大学);Institute of Scientific and Technical Research on Archives(档案科学技术研究所);China Electronics Engineering Design Institute Co., Ltd.(中国电子工程设计院有限公司)
Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reports
State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences(脑认知与脑启发智能技术重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);School of Future Technology, University of Chinese Academy of Sciences(未来技术学院,中国科学院大学);State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University(认知神经科学与学习国家重点实验室,北京师范大学)