arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.08495cs.CV
收藏

基于混合多模态融合的维度情感识别

Hybrid Mutimodal Fusion for Dimensional Emotion Recognition

Hunan University(湖南大学)

Ziyu Ma, Fuyan Ma, Bin Sun, Shutao Li

摘要:

在本文中,我们详细介绍了我们针对2021年多模态情感挑战(MuSe)的MuSe压力子挑战和MuSe生理子挑战的解决方案。MuSe-压力子挑战的目标是从视听记录中以时间连续的方式预测情绪唤醒和效价的水平,MuSe-生理子挑战的目标是从a)与b)来自压力人群的皮肤电反应(也称为皮肤电活动(EDA))信号融合的人类注释中预测心理生理唤醒的水平。Ulm-TSST数据集是视听文本Ulm-Trier Social Stress数据集的一个新子集,其特征在于在Trier Social Stress Test(TSST)诱导的压力情况下使用德语。对于MuSe-Stress子挑战,我们在三个方面突出了我们的解决方案:1)视听特征和生物信号特征用于情绪状态识别。2)利用具有自注意力机制的长短期记忆(LSTM)来捕获特征序列内的复杂时间依赖性。3)采用后期融合策略,充分利用多模态序列间的互补信息,进一步提高模型的识别性能。我们提出的模型在测试集上分别实现了效价和唤醒的CCC为0.6159和0.4609,这两个都排在前3位。对于MuSe-Physio子挑战,我们首先从多个模态中提取视听特征和生物信号特征。然后,利用具有自注意力机制的LSTM模块、门控卷积神经网络(GCNN)以及LSTM网络来对序列中的复杂时间依赖性进行建模。最后,采用了后期融合策略。我们提出的方法在测试集上也达到了0.5412的CCC,排名前3。

英文摘要:

In this paper, we extensively present our solutions for the MuSe-Stress sub-challenge and the MuSe-Physio sub-challenge of Multimodal Sentiment Challenge (MuSe) 2021. The goal of MuSe-Stress sub-challenge is to predict the level of emotional arousal and valence in a time-continuous manner from audio-visual recordings and the goal of MuSe-Physio sub-challenge is to predict the level of psycho-physiological arousal from a) human annotations fused with b) galvanic skin response (also known as Electrodermal Activity (EDA)) signals from the stressed people. The Ulm-TSST dataset which is a novel subset of the audio-visual textual Ulm-Trier Social Stress dataset that features German speakers in a Trier Social Stress Test (TSST) induced stress situation is used in both sub-challenges. For the MuSe-Stress sub-challenge, we highlight our solutions in three aspects: 1) the audio-visual features and the bio-signal features are used for emotional state recognition. 2) the Long Short-Term Memory (LSTM) with the self-attention mechanism is utilized to capture complex temporal dependencies within the feature sequences. 3) the late fusion strategy is adopted to further boost the model's recognition performance by exploiting complementary information scattered across multimodal sequences. Our proposed model achieves CCC of 0.6159 and 0.4609 for valence and arousal respectively on the test set, which both rank in the top 3. For the MuSe-Physio sub-challenge, we first extract the audio-visual features and the bio-signal features from multiple modalities. Then, the LSTM module with the self-attention mechanism, and the Gated Convolutional Neural Networks (GCNN) as well as the LSTM network are utilized for modeling the complex temporal dependencies in the sequence. Finally, the late fusion strategy is used. Our proposed method also achieves CCC of 0.5412 on the test set, which ranks in the top 3.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑