arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.00970cs.MM
收藏

通过超图自编码器和对比学习实现多模态融合的对话情感识别

Multimodal Fusion via Hypergraph Autoencoder and Contrastive Learning for Emotion Recognition in Conversation

Zijian Yi, Ziming Zhao, Zhishu Shen, Tiehua Zhang

AI总结:

针对MERC中固定全连接图导致的上下文冗余与过度平滑问题,提出利用VHGAE动态调整超图并结合对比学习的融合框架,在IEMOCAP和MELD上优于现有方法。

摘要:

对话中的多模态情感识别(MERC)旨在识别说话者在每个话语中表达的情感,在多个领域具有显著潜力。MERC的挑战在于平衡说话者建模与上下文建模,其中既包括长距离上下文也包括短距离上下文,同时还要应对多模态信息融合的复杂性。近期研究采用基于图的方法来有效建模复杂的对话关系。然而,这些方法大多使用固定的全连接结构连接所有话语,并依赖卷积来解释复杂上下文。这种方法固有地会加剧上下文消息中的冗余以及过度的图网络平滑,尤其是在长距离对话场景中。为解决该问题,我们提出一个框架,通过变分超图自编码器(VHGAE)动态调整超图连接,并采用对比学习来缓解重构过程中的不确定因素。实验结果表明,在IEMOCAP和MELD数据集上,我们的方案相较最先进方法具有有效性。我们在https://github.com/yzjred/-HAUCL发布代码以支持本工作的可复现性。

英文摘要:

Multimodal emotion recognition in conversation (MERC) seeks to identify the speakers' emotions expressed in each utterance, offering significant potential across diverse fields. The challenge of MERC lies in balancing speaker modeling and context modeling, encompassing both long-distance and short-distance contexts, as well as addressing the complexity of multimodal information fusion. Recent research adopts graph-based methods to model intricate conversational relationships effectively. Nevertheless, the majority of these methods utilize a fixed fully connected structure to link all utterances, relying on convolution to interpret complex context. This approach can inherently heighten the redundancy in contextual messages and excessive graph network smoothing, particularly in the context of long-distance conversations. To address this issue, we propose a framework that dynamically adjusts hypergraph connections by variational hypergraph autoencoder (VHGAE), and employs contrastive learning to mitigate uncertainty factors during the reconstruction process. Experimental results demonstrate the effectiveness of our proposal against the state-of-the-art methods on IEMOCAP and MELD datasets. We release the code to support the reproducibility of this work at https://github.com/yzjred/-HAUCL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑