arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.10201cs.MMcs.AI
收藏

CARAT:面向多模态多标签情感识别的对比特征重建与聚合

CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition

Zhejiang University(浙江大学)

Cheng Peng, Ke Chen, Lidan Shou, Gang Chen

AI总结:

本文提出 CARAT 方法,通过对比特征重建建模模态到标签的细粒度依赖,并利用打乱聚合策略增强标签共现协作,在 CMU-MOSEI 和 M3ED 上取得优于现有方法的多模态多标签情感识别效果。

摘要:

多模态多标签情感识别(MMER)旨在从多种模态中识别相关情感。MMER 的挑战在于如何从异构数据中有效捕获多个标签的判别性特征。近期研究主要致力于探索各种融合策略,将多模态信息整合为面向所有标签的统一表示。然而,这种学习方案不仅忽视了每种模态的特异性,而且未能捕获不同标签各自的判别性特征。此外,标签与模态之间的依赖关系也无法被有效建模。为解决这些问题,本文提出了用于 MMER 任务的 ContrAstive feature Reconstruction and AggregaTion(CARAT)。具体而言,我们设计了一种基于重建的融合机制,通过对比学习模态分离和标签特定特征,更好地建模细粒度的模态到标签依赖关系。为了进一步利用模态互补性,我们引入了一种基于打乱的聚合策略,以丰富标签之间的共现协作。在两个基准数据集 CMU-MOSEI 和 M3ED 上的实验证明了 CARAT 相对于最先进方法的有效性。代码可在 https://github.com/chengzju/CARAT 获取。

英文摘要:

Multi-modal multi-label emotion recognition (MMER) aims to identify relevant emotions from multiple modalities. The challenge of MMER is how to effectively capture discriminative features for multiple labels from heterogeneous data. Recent studies are mainly devoted to exploring various fusion strategies to integrate multi-modal information into a unified representation for all labels. However, such a learning scheme not only overlooks the specificity of each modality but also fails to capture individual discriminative features for different labels. Moreover, dependencies of labels and modalities cannot be effectively modeled. To address these issues, this paper presents ContrAstive feature Reconstruction and AggregaTion (CARAT) for the MMER task. Specifically, we devise a reconstruction-based fusion mechanism to better model fine-grained modality-to-label dependencies by contrastively learning modal-separated and label-specific features. To further exploit the modality complementarity, we introduce a shuffle-based aggregation strategy to enrich co-occurrence collaboration among labels. Experiments on two benchmark datasets CMU-MOSEI and M3ED demonstrate the effectiveness of CARAT over state-of-the-art methods. Code is available at https://github.com/chengzju/CARAT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑