arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.10335cs.MMcs.SDeess.AS
收藏

多模态连续情感识别:ABAW5技术报告

Multimodal Continuous Emotion Recognition: A Technical Report for ABAW5

Nanyang Technological University(南洋理工大学)

Su Zhang, Ziyuan Zhao, Cuntai Guan

AI总结:

本文提出两种多模态模型用于连续效价-唤醒度识别,融合视觉、音频和语言信息,通过通道注意力、对数梅尔频谱图微调VGG及交叉验证提升性能。

摘要:

我们使用两个多模态模型进行连续效价-唤醒度识别,利用视觉、音频和语言信息。第一个模型与我们在ABAW2和ABAW3中使用的相同,采用领导者-跟随者注意力机制。第二个模型在空间和时间编码上具有相同的架构。在融合模块方面,它采用了一个紧凑且直接的通道注意力机制,借鉴自End2You工具包。与我们之前直接使用Vggish特征作为音频特征的尝试不同,这次我们使用对数梅尔频谱图输入预训练的VGG模型,并在训练过程中对其进行微调。为了充分利用数据并缓解过拟合,进行了交叉验证。代码可在https://github.com/sucv/ABAW3获取。

英文摘要:

We used two multimodal models for continuous valence-arousal recognition using visual, audio, and linguistic information. The first model is the same as we used in ABAW2 and ABAW3, which employs the leader-follower attention. The second model has the same architecture for spatial and temporal encoding. As for the fusion block, it employs a compact and straightforward channel attention, borrowed from the End2You toolkit. Unlike our previous attempts that use Vggish feature directly as the audio feature, this time we feed the pre-trained VGG model using logmel-spectrogram and finetune it during the training. To make full use of the data and alleviate over-fitting, cross-validation is carried out. The code is available at https://github.com/sucv/ABAW3.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑