arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.00547cs.CVcs.AIcs.HC
收藏

Milmer: 一种基于多实例学习的多模态情感识别框架

Milmer: a Framework for Multiple Instance Learning based Multimodal Emotion Recognition

Zaitian Wang, Jian He, Yu Liang, Xiyuan Hu, Tianhao Peng, Kaixin Wang, Jiakai Wang, Chenlong Zhang, Weili Zhang, Shuang Niu, Xiaoyang Xie

AI总结:

本文提出Milmer框架,结合面部表情分析与EEG信号,采用Transformer融合方法提升多模态情感识别性能,通过细调Swin Transformer和交叉注意力机制实现高效特征融合,实验显示在DEAP数据集上达到96.72%的四类情感识别准确率。

摘要:

情感在人类行为和决策中起关键作用,是人机交互(HCI)的重要研究领域。本文通过整合面部表情分析与脑电图(EEG)信号,提出一种新的多模态框架-Milmer。该框架采用基于Transformer的融合方法,包含EEG预处理模块、面部特征提取与平衡模块以及跨模态融合模块。为提升视觉特征提取,我们对预训练的Swin Transformer进行微调。此外,引入交叉注意力机制以平衡跨模态的token表示,确保有效特征整合。本文的关键创新是采用多实例学习(MIL)方法,从多个面部表情图像中提取时间动态信息。在DEAP数据集上的大量实验验证了所提框架的优越性,实现了四类情感识别任务的96.72%分类准确率。消融研究进一步验证了各模块的贡献,突显了先进特征提取和融合策略在提升情感识别性能中的重要性。代码已开源于https://github.com/liangyubuaa/Milmer。

英文摘要:

Emotions play a crucial role in human behavior and decision-making, making emotion recognition a key area of interest in human-computer interaction (HCI). This study addresses the challenges of emotion recognition by integrating facial expression analysis with electroencephalogram (EEG) signals, introducing a novel multimodal framework-Milmer. The proposed framework employs a transformer-based fusion approach to effectively integrate visual and physiological modalities. It consists of an EEG preprocessing module, a facial feature extraction and balancing module, and a cross-modal fusion module. To enhance visual feature extraction, we fine-tune a pre-trained Swin Transformer on emotion-related datasets. Additionally, a cross-attention mechanism is introduced to balance token representation across modalities, ensuring effective feature integration. A key innovation of this work is the adoption of a multiple instance learning (MIL) approach, which extracts meaningful information from multiple facial expression images over time, capturing critical temporal dynamics often overlooked in previous studies. Extensive experiments conducted on the DEAP dataset demonstrate the superiority of the proposed framework, achieving a classification accuracy of 96.72% in the four-class emotion recognition task. Ablation studies further validate the contributions of each module, highlighting the significance of advanced feature extraction and fusion strategies in enhancing emotion recognition performance. Our code are available at https://github.com/liangyubuaa/Milmer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑