arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.12895cs.MM
收藏

Ada2I:增强多模态对话情感识别的模态平衡

Ada2I: Enhancing Modality Balance for Multimodal Conversational Emotion Recognition

Cam-Van Thi Nguyen, The-Son Le, Anh-Tuan Mai, Duc-Trong Le

AI总结:

针对多模态对话情感识别中的模态不平衡问题,提出Ada2I框架,通过自适应特征与模态加权模块结合改进的差异比率优化策略,在三个基准数据集上实现了最先进的性能。

摘要:

对话中的多模态情感识别(ERC)是一项典型的同时利用各种数据模态的多模态学习任务。先前关于有效多模态ERC的研究在解决模态不平衡和优化跨模态学习方面面临挑战。针对这些问题,我们提出了一个名为Ada2I的新框架,它包含两个不可分割的模块,即自适应特征加权(AFW)和自适应模态加权(AMW),通过利用模态间和模态内交互分别实现特征级和模态级的平衡。此外,我们引入了改进的差异比率作为训练优化策略的一部分,这是一种简单而有效的度量方法,用于评估模型在同时处理多个模态时学习过程的整体差异。实验结果验证了Ada2I的有效性,与三个基准数据集上的基线相比取得了最先进的性能,特别是在解决模态不平衡方面。

英文摘要:

Multimodal Emotion Recognition in Conversations (ERC) is a typical multimodal learning task in exploiting various data modalities concurrently. Prior studies on effective multimodal ERC encounter challenges in addressing modality imbalances and optimizing learning across modalities. Dealing with these problems, we present a novel framework named Ada2I, which consists of two inseparable modules namely Adaptive Feature Weighting (AFW) and Adaptive Modality Weighting (AMW) for feature-level and modality-level balancing respectively via leveraging both Inter- and Intra-modal interactions. Additionally, we introduce a refined disparity ratio as part of our training optimization strategy, a simple yet effective measure to assess the overall discrepancy of the model's learning process when handling multiple modalities simultaneously. Experimental results validate the effectiveness of Ada2I with state-of-the-art performance compared to baselines on three benchmark datasets, particularly in addressing modality imbalances.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑