通过狄利克雷参数化实现的不确定性感知多模态情绪识别
Uncertainty-Aware Multimodal Emotion Recognition through Dirichlet Parameterization
AI总结:
本文提出一种轻量级且隐私保护的多模态情绪识别框架,通过狄利克雷参数化方法协调不同模态的不确定性,实现高效且鲁棒的情绪识别。
摘要:
在本工作中,我们提出了一种轻量级且隐私保护的多模态情绪识别(MER)框架,适用于边缘设备部署。为了展示框架的通用性,我们的实现使用了三种模态——语音、文本和面部图像。然而,该系统是完全模块化的,可以扩展以支持其他模态或任务。每种模态都通过专用的骨干网络进行处理,以优化推理效率:Emotion2Vec用于语音,基于ResNet的模型用于面部表情,DistilRoBERTa用于文本。为了在不同模态之间协调不确定性,我们引入了一种基于Dempster-Shafer理论和狄利克雷证据的模型和任务无关的融合机制。该方法直接在模型logits上操作,能够捕捉预测不确定性,而无需额外训练或联合分布估计,使其在情绪识别之外的应用也具有广泛适用性。在五个基准数据集(eNTERFACE05、MEAD、MELD、RAVDESS和CREMA-D)上的验证表明,我们的方法在保持计算效率和对模糊或缺失输入的鲁棒性的同时,实现了具有竞争力的准确性。总体而言,所提出的框架强调了模块化、可扩展性和现实可行性,为医疗、人机交互和其他情绪导向应用中的不确定性感知多模态系统铺平了道路。
英文摘要:
In this work, we present a lightweight and privacy-preserving Multimodal Emotion Recognition (MER) framework designed for deployment on edge devices. To demonstrate framework's versatility, our implementation uses three modalities - speech, text and facial imagery. However, the system is fully modular, and can be extended to support other modalities or tasks. Each modality is processed through a dedicated backbone optimized for inference efficiency: Emotion2Vec for speech, a ResNet-based model for facial expressions, and DistilRoBERTa for text. To reconcile uncertainty across modalities, we introduce a model- and task-agnostic fusion mechanism grounded in Dempster-Shafer theory and Dirichlet evidence. Operating directly on model logits, this approach captures predictive uncertainty without requiring additional training or joint distribution estimation, making it broadly applicable beyond emotion recognition. Validation on five benchmark datasets (eNTERFACE05, MEAD, MELD, RAVDESS and CREMA-D) show that our method achieves competitive accuracy while remaining computationally efficient and robust to ambiguous or missing inputs. Overall, the proposed framework emphasizes modularity, scalability, and real-world feasibility, paving the way toward uncertainty-aware multimodal systems for healthcare, human-computer interaction, and other emotion-informed applications.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。