arXiv 2409.050152024-09-10cs.HCcs.SDeess.AS

利用声学适配与视觉对齐改进多模态情感识别

Improving Multimodal Emotion Recognition by Leveraging Acoustic Adaptation and Visual Alignment

Northwestern Polytechnical University（西北工业大学）；Shaanxi University of Science and Technology（陕西科技大学）；Peng Cheng Laboratory（鹏城实验室）

Zhixian Zhao, Haifeng Chen, Xi Li, Dongmei Jiang, Lei Xie

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

面向 MER2024-SEMI 半监督多模态情感识别任务，提出 HuBERT 声学层 PEFT 适配、视觉特征向声学空间对齐预训练及注意力融合方法，加权 F1 达 88.90%，排名第四。

摘要：

多模态情感识别（Multimodal Emotion Recognition，MER）旨在通过整合来自多种模态的信息，自动识别和理解人类的情感状态。然而，带标注的多模态数据稀缺，严重阻碍了该研究领域的进展。本文介绍了我们针对 MER 2024 的 MER-SEMI 子挑战赛提出的解决方案。首先，为了使声学模态特征更好地适配 MER 任务，我们通过实验评估了预训练语音模型 HuBERT 的不同层在情感识别中的贡献。基于这些观察，我们对被识别为对情感识别任务最有效的层执行参数高效微调（Parameter-Efficient Fine-Tuning，PEFT），从而以最少的可学习参数实现最佳的情感识别适配。其次，利用声学模态的优势，我们提出了一种特征对齐预训练方法。该方法使用大规模无标注数据训练视觉编码器，从而促进视觉特征在声学特征空间内的语义对齐。最后，我们使用适配后的声学特征、对齐后的视觉特征以及词汇特征，并采用注意力机制进行特征融合。在 MER2024-SEMI 测试集上，所提方法取得了 88.90% 的加权 F1 分数，在所有参赛队伍中排名第四，验证了我们方法的有效性。

英文摘要：

Multimodal Emotion Recognition (MER) aims to automatically identify and understand human emotional states by integrating information from various modalities. However, the scarcity of annotated multimodal data significantly hinders the advancement of this research field. This paper presents our solution for the MER-SEMI sub-challenge of MER 2024. First, to better adapt acoustic modality features for the MER task, we experimentally evaluate the contributions of different layers of the pre-trained speech model HuBERT in emotion recognition. Based on these observations, we perform Parameter-Efficient Fine-Tuning (PEFT) on the layers identified as most effective for emotion recognition tasks, thereby achieving optimal adaptation for emotion recognition with a minimal number of learnable parameters. Second, leveraging the strengths of the acoustic modality, we propose a feature alignment pre-training method. This approach uses large-scale unlabeled data to train a visual encoder, thereby promoting the semantic alignment of visual features within the acoustic feature space. Finally, using the adapted acoustic features, aligned visual features, and lexical features, we employ an attention mechanism for feature fusion. On the MER2024-SEMI test set, the proposed method achieves a weighted F1 score of 88.90%, ranking fourth among all participating teams, validating the effectiveness of our approach.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。