基于多通道和对称互促特征融合的多模态情感分析
Multimodal Sentiment Analysis based on Multi-channel and Symmetric Mutual Promotion Feature Fusion
AI总结：
本文提出基于多通道和对称互促特征融合的多模态情感分析方法，通过增强模态内特征表示和促进模态间信息交互，提升情感识别的准确性和鲁棒性。
摘要：
多模态情感分析是人机交互和情感计算领域的重要技术。准确识别人类情绪状态对于促进人机之间顺畅交流至关重要。尽管在多模态情感分析研究中取得了一些进展，但仍存在诸多挑战。第一个挑战是单模态数据中提取的特征有限且不够丰富。其次，大多数研究只关注模态间特征信息的一致性，忽视了特征之间的差异，导致特征信息融合不足。在本文中，我们首先提取多通道特征以获得更全面的特征信息。我们采用视觉和听觉模态中的双通道特征以增强模态内特征表示。其次，我们提出了一种对称互促（SMP）的模态间特征融合方法。该方法结合了对称跨模态注意力机制和自注意力机制，其中跨模态注意力机制从其他模态中捕捉有用信息，自注意力机制建模上下文信息。这种方法促进了模态间有用信息的交流，从而加强了模态间交互。此外，我们整合了模态内特征和模态间融合的特征，充分利用了模态间特征信息的互补性，同时考虑了特征信息的差异。在两个基准数据集上进行的实验证明了我们所提出的方法的有效性和优越性。
英文摘要：
Multimodal sentiment analysis is a key technology in the fields of human-computer interaction and affective computing. Accurately recognizing human emotional states is crucial for facilitating smooth communication between humans and machines. Despite some progress in multimodal sentiment analysis research, numerous challenges remain. The first challenge is the limited and insufficiently rich features extracted from single modality data. Secondly, most studies focus only on the consistency of inter-modal feature information, neglecting the differences between features, resulting in inadequate feature information fusion. In this paper, we first extract multi-channel features to obtain more comprehensive feature information. We employ dual-channel features in both the visual and auditory modalities to enhance intra-modal feature representation. Secondly, we propose a symmetric mutual promotion (SMP) inter-modal feature fusion method. This method combines symmetric cross-modal attention mechanisms and self-attention mechanisms, where the cross-modal attention mechanism captures useful information from other modalities, and the self-attention mechanism models contextual information. This approach promotes the exchange of useful information between modalities, thereby strengthening inter-modal interactions. Furthermore, we integrate intra-modal features and inter-modal fused features, fully leveraging the complementarity of inter-modal feature information while considering feature information differences. Experiments conducted on two benchmark datasets demonstrate the effectiveness and superiority of our proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
