在本文中,我们详细介绍了我们针对2021年多模态情感挑战(MuSe)的MuSe压力子挑战和MuSe生理子挑战的解决方案。MuSe-压力子挑战的目标是从视听记录中以时间连续的方式预测情绪唤醒和效价的水平,MuSe-生理子挑战的目标是从a)与b)来自压力人群的皮肤电反应(也称为皮肤电活动(EDA))信号融合的人类注释中预测心理生理唤醒的水平。Ulm-TSST数据集是视听文本Ulm-Trier Social Stress数据集的一个新子集,其特征在于在Trier Social Stress Test(TSST)诱导的压力情况下使用德语。对于MuSe-Stress子挑战,我们在三个方面突出了我们的解决方案:1)视听特征和生物信号特征用于情绪状态识别。2)利用具有自注意力机制的长短期记忆(LSTM)来捕获特征序列内的复杂时间依赖性。3)采用后期融合策略,充分利用多模态序列间的互补信息,进一步提高模型的识别性能。我们提出的模型在测试集上分别实现了效价和唤醒的CCC为0.6159和0.4609,这两个都排在前3位。对于MuSe-Physio子挑战,我们首先从多个模态中提取视听特征和生物信号特征。然后,利用具有自注意力机制的LSTM模块、门控卷积神经网络(GCNN)以及LSTM网络来对序列中的复杂时间依赖性进行建模。最后,采用了后期融合策略。我们提出的方法在测试集上也达到了0.5412的CCC,排名前3。
英文摘要:
In this paper, we extensively present our solutions for the MuSe-Stress sub-challenge and the MuSe-Physio sub-challenge of Multimodal Sentiment Challenge (MuSe) 2021. The goal of MuSe-Stress sub-challenge is to predict the level of emotional arousal and valence in a time-continuous manner from audio-visual recordings and the goal of MuSe-Physio sub-challenge is to predict the level of psycho-physiological arousal from a) human annotations fused with b) galvanic skin response (also known as Electrodermal Activity (EDA)) signals from the stressed people. The Ulm-TSST dataset which is a novel subset of the audio-visual textual Ulm-Trier Social Stress dataset that features German speakers in a Trier Social Stress Test (TSST) induced stress situation is used in both sub-challenges. For the MuSe-Stress sub-challenge, we highlight our solutions in three aspects: 1) the audio-visual features and the bio-signal features are used for emotional state recognition. 2) the Long Short-Term Memory (LSTM) with the self-attention mechanism is utilized to capture complex temporal dependencies within the feature sequences. 3) the late fusion strategy is adopted to further boost the model's recognition performance by exploiting complementary information scattered across multimodal sequences. Our proposed model achieves CCC of 0.6159 and 0.4609 for valence and arousal respectively on the test set, which both rank in the top 3. For the MuSe-Physio sub-challenge, we first extract the audio-visual features and the bio-signal features from multiple modalities. Then, the LSTM module with the self-attention mechanism, and the Gated Convolutional Neural Networks (GCNN) as well as the LSTM network are utilized for modeling the complex temporal dependencies in the sequence. Finally, the late fusion strategy is used. Our proposed method also achieves CCC of 0.5412 on the test set, which ranks in the top 3.
Feature-level and Model-level Audiovisual Fusion for Emotion Recognition in the Wild
University of South Carolina(南卡罗来纳大学);Innopeak Technology Inc.(Innopeak科技公司);Sigma Technologies(12西格玛科技公司);JD.com, Inc.(京东集团);University of Washington Bothell(华盛顿大学博塞尔分校)
eMotions: A Large-Scale Dataset and Audio-Visual Fusion Network for Emotion Analysis in Short-form Videos
School of Computer Science and Technology, Xi’an Jiaotong University(计算机科学与技术学院,西安交通大学);College of Intelligent Robotics and Advanced Manufacturing, Fudan University & ByteDance(智能机器人与先进制造学院,复旦大学 & 字节跳动);School of Cyber Science and Engineering, Zhengzhou University(网络科学与工程学院,郑州大学);Institute of Advanced Technology, University of Science and Technology of China(先进技术研究院,中国科学技术大学);Inspur Electronic Information Industry Co., Ltd(Inspur电子信息产业有限公司);Department of Computer Science, The University of Toronto(计算机科学系,多伦多大学);Research Center for Space Computing System, Zhejiang Lab(空间计算系统研究中心,浙江实验室);Institute of Information Technology, University of Klagenfurt(信息技术学院,克雷格福特大学);School of Computer Science and Informatics, Cardiff University(计算机科学与信息学院,卡迪夫大学)
Smile on the Face, Sadness in the Eyes: Bridging the Emotion Gap with a Multimodal Dataset of Eye and Facial Behaviors
School of Computer Science, China University of Geosciences (Wuhan)(中国地质大学(武汉)计算机科学学院);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件工程学院);School of Computer Science, Wuhan University(武汉大学计算机科学学院);School of Computing, Engineering and Mathematical Sciences, La Trobe University(拉筹伯大学计算科学、工程与数学科学学院)
Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition
Department of Biomedical Engineering(生物医学工程系);Anhui Province Key Laboratory of Measuring Theory and Precision Instrument(安徽省测量理论与精密仪器重点实验室);School of Computer Science and Information Engineering(计算机科学与信息工程学院)
Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition
自动情感识别是一项具有挑战性的任务。在本文中,我们介绍了我们在2018年野外情绪识别(Emotion Recognition in the Wild,EMOTIW)挑战赛中基于音频视频的子挑战赛的努力,该挑战赛要求参与者从六种普遍情绪(愤怒,厌恶,恐惧,快乐,悲伤和惊讶)和中性中为视频片段分配一个单一的情绪标签。所提出的多模态情感识别系统将音频、视频和文本信息考虑在内。除了手工特征,我们还通过迁移学习从深度神经网络(DNN)中提取瓶颈特征。对时间分类器和非时间分类器进行评估,以获得最佳的单峰情感分类结果。然后提取可能性并将其传递到束搜索融合(BS-融合)中。我们在2018年的挑战赛中测试了我们的方法,并获得了令人鼓舞的结果。与基线系统相比,有了明显的改进。我们在测试数据集上实现了60.34%的准确率,仅比获胜者低1.5%。这表明我们的方法非常有竞争力。
Single-Stream Multi-Feature Fusion with Temporal Robustness for Gait Emotion Recognition
College of Computer and Information Science, Southwest University(西南大学计算机与信息科学学院);College of Physical Science and Technology, Southwest University(西南大学物理科学与技术学院);Wisesoft Inc.(四川川大智胜软件股份有限公司)
本文讨论了情感分类的问题。该任务包括预测最能描述短视频剪辑中包含的情绪的情绪标签(从一组可能的标签中选取)。建立在一个标准的框架-躺在描述视频的音频和视觉特征使用的监督分类器来推断标签-本文研究了几个新的方向。首先,提出了基于二维和三维卷积神经网络的改进人脸描述子。其次,本文探讨了几种融合方法,时间和多模态,包括一种新的分层方法相结合的功能和分数。此外,我们仔细审查了流水线的不同阶段,并设计了一个适应任务的CNN架构;这一点很重要,因为与问题的难度相比,训练集的大小很小,使得泛化困难。该模型在2017年的Emotion in the Wild挑战赛中排名第四,准确率为58.8%。
DRKF: Decoupled Representations with Knowledge Fusion for Multimodal Emotion Recognition
School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院)
Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University(吉林大学符号计算与知识工程教育部重点实验室)
A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition
Beijing University of Posts and Telecommunications(北京邮电大学);East China Normal University(华东师范大学);Shanghai University of International Business and Economics(上海对外经贸大学)
Two-Aspect Information Fusion Model For ABAW4 Multi-task Challenge
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
TACFN: Transformer-based Adaptive Cross-modal Fusion Network for Multimodal Emotion Recognition
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);MTlab, Meitu (China) Limited(美图(中国)有限公司);Institute of Acoustics, University of Chinese Academy of Sciences(中国科学院声学研究所)
Multimodal Video Emotion Recognition with Reliable Reasoning Priors
Lenovo Research(联想研究院);School of Artificial Intelligence, UCAS(中国科学院大学人工智能学院);Institute of Automation, CAS(中国科学院自动化研究所);Macau University of Science and Technology(澳门科学理工学院);School of Computer Science and Technology, UCAS(中国科学院大学计算机科学与技术学院)