通过专家引导的多模态融合与大语言模型的统一框架实现情绪识别与情感分析
Expert-Guided Multimodal Fusion for Unified Emotion and Sentiment Analysis
AI总结：
本文提出EGMF框架，通过专家引导的多模态融合与大语言模型结合，实现情绪识别与情感分析的统一处理，提升跨语言鲁棒性。
摘要：
多模态情绪理解需要有效整合文本、音频和视觉模态，以实现离散情绪识别和连续情感分析。我们提出了EGMF，一种结合专家引导的多模态融合与大语言模型的统一框架。我们的方法包含三个专门的专家网络——一个细粒度局部专家用于捕捉细微的情绪特征，一个语义相关性专家用于处理跨模态关系，以及一个全局上下文专家用于处理长距离依赖——通过层次动态门控机制进行自适应整合。增强的多模态表示通过伪令牌注入和基于提示的条件化与大语言模型结合，使单个生成框架能够通过自然语言生成处理分类和回归任务。我们采用LoRA微调以提高计算效率。在双语基准（MELD、CHERMA、MOSEI、SIMS-V2）上的实验表明，与最先进方法相比，有持续改进，且在跨语言鲁棒性方面表现优异，揭示了英语和中文中多模态情绪表达中的普遍模式。我们将公开发布源代码。
英文摘要：
Multimodal emotion understanding requires the integration of heterogeneous data sources, including text, audio, and visual modalities, while simultaneously addressing discrete emotion recognition and continuous sentiment analysis. We propose EGMF, a unified framework that combines expert-guided multimodal fusion with large language models to achieve superior performance across both tasks. At the core of our framework is a multi-scale expert network, comprising a local expert for capturing subtle emotional nuances, a semantic correlation expert for modeling cross-modal relationships, and a global context expert for understanding long-range dependencies. These experts are adaptively integrated via hierarchical dynamic gating, enabling context-aware feature selection and modality weighting. The enhanced multimodal representations are seamlessly incorporated into the language model through pseudo token injection and prompt-based conditioning, allowing a single generative framework to handle both classification and regression tasks. We employ parameter-efficient LoRA fine-tuning to maintain computational efficiency. Extensive experiments on bilingual benchmark datasets (MELD, CHERMA, MOSEI, SIMS-V2) demonstrate that EGMF outperforms state-of-the-art methods in terms of accuracy, cross-lingual robustness, and the discovery of universal patterns in multimodal emotional expressions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
