arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2111.08910cs.SDcs.MMeess.AS
收藏

基于自适应多级分解双线性池的注意力网络信息融合视听情感识别

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

University of Science and Technology of China(中国科学技术大学);Georgia Institute of Technology(佐治亚理工学院)

Hengshun Zhou, Jun Du, Yuanyuan Zhang, Qing Wang, Qing-Feng Liu, Chin-Hui Lee

摘要:

多模态情感识别是情感计算中的一项挑战性任务,因为情感概念抽象,表达方式多样,很难提取具有区分性的特征来识别情感中的细微差别。此外,如何充分利用音频和视频信息仍然是一个悬而未决的问题。本文提出了一种基于自适应和多级分解双线性池(FBP)的多模态融合注意力网络用于视听情感识别。首先,针对音频流,设计了一种具有一维注意力机制和局部响应归一化的全卷积网络(FCN)用于语音情感识别。其次,提出了一种全局FBP(G-FBP)方法,通过将基于自注意的视频流与音频流相结合来实现视听信息融合。为了改进G-FBP算法,提出了一种自适应融合策略(AG-FBP),该策略根据不同模态的注意力机制得到的情感相关表征向量,动态计算两种模态的融合权值。最后,为了充分利用本地情感信息,在AG-FBP的基础上,引入了自适应多级FBP(AMFBP),将全局主干数据和主干内数据结合在一个记录中。在IEMOCAP语音情感识别语料库上进行测试,新的FCN方法优于最先进的结果,准确率为71.40%。此外,在AFEW数据库上进行了验证2019 sub-challenge和用于视听情感识别的IEMOCAP语料库,所提出的AM-FBP方法在测试集上分别达到了63.09%和75.49%的最佳准确率。

英文摘要:

Multimodal emotion recognition is a challenging task in emotion computing as it is quite difficult to extract discriminative features to identify the subtle differences in human emotions with abstract concept and multiple expressions. Moreover, how to fully utilize both audio and visual information is still an open problem. In this paper, we propose a novel multimodal fusion attention network for audio-visual emotion recognition based on adaptive and multi-level factorized bilinear pooling (FBP). First, for the audio stream, a fully convolutional network (FCN) equipped with 1-D attention mechanism and local response normalization is designed for speech emotion recognition. Next, a global FBP (G-FBP) approach is presented to perform audio-visual information fusion by integrating selfattention based video stream with the proposed audio stream. To improve G-FBP, an adaptive strategy (AG-FBP) to dynamically calculate the fusion weight of two modalities is devised based on the emotion-related representation vectors from the attention mechanism of respective modalities. Finally, to fully utilize the local emotion information, adaptive and multi-level FBP (AMFBP) is introduced by combining both global-trunk and intratrunk data in one recording on top of AG-FBP. Tested on the IEMOCAP corpus for speech emotion recognition with only audio stream, the new FCN method outperforms the state-ofthe-art results with an accuracy of 71.40%. Moreover, validated on the AFEW database of EmotiW2019 sub-challenge and the IEMOCAP corpus for audio-visual emotion recognition, the proposed AM-FBP approach achieves the best accuracy of 63.09% and 75.49% respectively on the test set.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑