EMERSK——基于情境知识的可解释多模态情感识别
EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge
AI总结：
针对情感识别中多模态利用与结果可解释性的挑战，提出模块化多模态系统EMERSK，融合多模态视觉特征与情境知识实现情感识别与解释，经实验验证性能优于现有最优方法。
摘要：
近年来，随着深度学习算法的日益普及，自动情感识别受到了广泛关注。情感识别的主要挑战之一是有效利用数据中可用的各种线索（模态），另一项挑战是为学习结果提供合理的解释。为应对这些挑战，我们提出了基于情境知识的可解释多模态情感识别系统（Explainable Multimodal Emotion Recognition with Situational Knowledge，EMERSK），这是一个通用模块化系统，可利用视觉信息进行人类情感识别与解释。我们的系统能够以灵活模块化的方式处理多种模态，包括面部表情、姿态和步态。该网络由不同模块组成，可根据可用数据添加或移除模块。我们采用双流网络架构，结合卷积神经网络（CNNs）与编解码器风格的注意力机制，从人脸图像中提取深度特征；同样地，我们使用带有长短期记忆（LSTM）的卷积神经网络（CNNs）和循环神经网络（RNNs），从姿态和步态数据中提取特征。我们还将来自背景的深度特征作为上下文信息纳入学习过程，各模块的深度特征通过早期融合网络进行融合。此外，我们利用从场景中提取的位置类型、形容词-名词对（ANP）以及情感的时空平均分布得到的情境知识，生成解释。消融实验表明，每个子网络都能独立完成情感识别，而采用多模态方式将它们结合后，整体识别性能得到显著提升。我们在包括GroupWalk在内的多个基准数据集上开展了大量实验，结果验证了我们的方法相较于其他现有最优方法具有更优的性能。
英文摘要：
Automatic emotion recognition has recently gained significant attention due to the growing popularity of deep learning algorithms. One of the primary challenges in emotion recognition is effectively utilizing the various cues (modalities) available in the data. Another challenge is providing a proper explanation of the outcome of the learning.To address these challenges, we present Explainable Multimodal Emotion Recognition with Situational Knowledge (EMERSK), a generalized and modular system for human emotion recognition and explanation using visual information. Our system can handle multiple modalities, including facial expressions, posture, and gait, in a flexible and modular manner. The network consists of different modules that can be added or removed depending on the available data. We utilize a two-stream network architecture with convolutional neural networks (CNNs) and encoder-decoder style attention mechanisms to extract deep features from face images. Similarly, CNNs and recurrent neural networks (RNNs) with Long Short-term Memory (LSTM) are employed to extract features from posture and gait data. We also incorporate deep features from the background as contextual information for the learning process. The deep features from each module are fused using an early fusion network. Furthermore, we leverage situational knowledge derived from the location type and adjective-noun pair (ANP) extracted from the scene, as well as the spatio-temporal average distribution of emotions, to generate explanations. Ablation studies demonstrate that each sub-network can independently perform emotion recognition, and combining them in a multimodal approach significantly improves overall recognition performance. Extensive experiments conducted on various benchmark datasets, including GroupWalk, validate the superior performance of our approach compared to other state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
