arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.23300cs.CLeess.AS
收藏

一种用于对话中多模态情绪识别的专家混合模型

A Mixture-of-Experts Model for Multimodal Emotion Recognition in Conversations

LEAP Lab, Department of Electrical Engineering(LEAP实验室,电气工程系);Microsoft(微软)

Soumya Dutta, Smruthi Balaji, Sriram Ganapathy

AI总结:

MiSTER-E通过专家混合框架提升对话中多模态情绪识别的准确率,实现跨模态一致性与融合。

摘要:

对话中的情绪识别(ERC)提出了独特的挑战,要求模型能够捕捉多轮对话的时间流动,并有效整合多种模态的线索。我们提出了MiSTER-E,一种模块化的专家混合(MoE)框架,旨在分离ERC中的两个核心挑战:模态特定的上下文建模和多模态信息融合。MiSTER-E利用经过语音和文本微调的大语言模型(LLMs)提供丰富的语句级嵌入,然后通过卷积-循环上下文建模层进行增强。系统整合语音-only、文本-only和跨模态三个专家的预测,使用学习到的门控机制动态加权其输出。为了进一步鼓励模态之间的一致性和对齐,我们引入了配对语音-文本表示之间的监督对比损失以及专家预测之间的KL散度基于正则化。重要的是,MiSTER-E在任何阶段都不依赖说话人身份。在三个基准数据集-IEMOCAP、MELD和MOSI上的实验表明,我们的方法分别达到了70.9%、69.5%和87.9%的加权F1分数,优于几种基线语音-文本ERC系统。我们还提供了各种消融实验以突出所提出方法的贡献。

英文摘要:

Emotion Recognition in Conversations (ERC) presents unique challenges, requiring models to capture the temporal flow of multi-turn dialogues and to effectively integrate cues from multiple modalities. We propose Mixture of Speech-Text Experts for Recognition of Emotions (MiSTER-E), a modular Mixture-of-Experts (MoE) framework designed to decouple two core challenges in ERC: modality-specific context modeling and multimodal information fusion. MiSTER-E leverages large language models (LLMs) fine-tuned for both speech and text to provide rich utterance-level embeddings, which are then enhanced through a convolutional-recurrent context modeling layer. The system integrates predictions from three experts-speech-only, text-only, and cross-modal-using a learned gating mechanism that dynamically weighs their outputs. To further encourage consistency and alignment across modalities, we introduce a supervised contrastive loss between paired speech-text representations and a KL-divergence-based regulariza-tion across expert predictions. Importantly, MiSTER-E does not rely on speaker identity at any stage. Experiments on three benchmark datasets-IEMOCAP, MELD, and MOSI-show that our proposal achieves 70.9%, 69.5%, and 87.9% weighted F1-scores respectively, outperforming several baseline speech-text ERC systems. We also provide various ablations to highlight the contributions made in the proposed approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑