arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.02845cs.AI
收藏

利用面部特征和生理信号的可解释多模态情感识别

Interpretable Multimodal Emotion Recognition using Facial Features and Physiological Signals

University of Oulu(奥卢大学)

Puneet Kumar, Xiaobai Li

AI总结:

提出融合视觉面部特征与视频提取rPPG信号的多模态情感识别框架,并用排列特征重要性解释各模态贡献,在IEMOCAP上验证了融合可提升分类性能。

摘要:

本文旨在证明融合多模态信息用于情感识别的重要性和可行性。文中提出一种用于情感理解的多模态框架,通过融合视觉面部特征与从输入视频中提取的rPPG信号的信息。研究还实现了一种基于排列特征重要性分析的可解释性技术,用以计算rPPG和视觉模态在将给定输入视频分类为特定情感类别时的贡献。在IEMOCAP数据集上的实验表明,通过结合来自多种模态的互补信息,情感分类性能得到提升。

英文摘要:

This paper aims to demonstrate the importance and feasibility of fusing multimodal information for emotion recognition. It introduces a multimodal framework for emotion understanding by fusing the information from visual facial features and rPPG signals extracted from the input videos. An interpretability technique based on permutation feature importance analysis has also been implemented to compute the contributions of rPPG and visual modalities toward classifying a given input video into a particular emotion class. The experiments on IEMOCAP dataset demonstrate that the emotion classification performance improves by combining the complementary information from multiple modalities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑