This paper aims to demonstrate the importance and feasibility of fusing multimodal information for emotion recognition. It introduces a multimodal framework for emotion understanding by fusing the information from visual facial features and rPPG signals extracted from the input videos. An interpretability technique based on permutation feature importance analysis has also been implemented to compute the contributions of rPPG and visual modalities toward classifying a given input video into a particular emotion class. The experiments on IEMOCAP dataset demonstrate that the emotion classification performance improves by combining the complementary information from multiple modalities.
The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress
University of Augsburg(奥格斯堡大学);University of Ulm(乌尔姆大学);Nanyang Technological University(南洋理工大学);University of Oulu(奥卢大学);Imperial College London(帝国理工学院)
ASLSL: Adaptive shared latent structure learning with incomplete multi-modal physiological data for multi-dimensional emotional feature selection
School of Information Science and Technology, Beijing University of Technology, Beijing 100124, China(信息科学与技术学院,北京理工大学,北京);School of Artificial Intelligence, Anhui University, Beijing 100124, China(人工智能学院,安徽大学,北京)
Multi-Masked Querying Network for Robust Emotion Recognition from Incomplete Multi-Modal Physiological Signals
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China(深圳先进技术研究院,中国科学院,深圳518055,中国);Southern University of Science and Technology, Shenzhen 518055, China(南方科技大学,深圳518055,中国)
PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation
MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS部);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Psyche AI.INC(Psyche AI公司);HKUST(香港科技大学);CAIR, HKISI, Chinese Academy of Sciences(中国科学院计算智能研究所);SCSE, FIE, M.U.S.T(M.U.S.T的SCSE、FIE部门)
Smile on the Face, Sadness in the Eyes: Bridging the Emotion Gap with a Multimodal Dataset of Eye and Facial Behaviors
School of Computer Science, China University of Geosciences (Wuhan)(中国地质大学(武汉)计算机科学学院);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件工程学院);School of Computer Science, Wuhan University(武汉大学计算机科学学院);School of Computing, Engineering and Mathematical Sciences, La Trobe University(拉筹伯大学计算科学、工程与数学科学学院)
Controllable Expressive 3D Facial Animation via Diffusion in a Unified Multimodal Space
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Laboratory for Big Data and Decision, School of Systems Engineering, National University of Defense Technology(国防科技大学系统工程学院大数据与决策实验室)