MicroEmo：视频对话中基于微表情动力学的时间敏感多模态情感识别
MicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialogues
AI总结：
提出时间敏感模型MicroEmo，通过全局-局部注意力视觉编码器和话语感知视频Q-Former捕捉微表情动力学与上下文依赖，在可解释多模态情感识别任务中验证了有效性。
摘要：
多模态大语言模型（MLLMs）已展现出卓越的多模态情感识别能力，通过整合视频中视觉、声学和语言语境的多模态线索来识别人类情感状态。然而，现有方法忽略了捕捉微表情时间动力学的局部面部特征，并未利用视频中话语感知时间片段的上下文依赖关系，从而在一定程度上限制了其预期效果。在这项工作中，我们提出了 MicroEmo，这是一种时间敏感的 MLLM，旨在将注意力引向局部面部微表情动力学和话语感知视频片段的上下文依赖关系。我们的模型包含两项关键的架构贡献：（1）全局-局部注意力视觉编码器，将时间戳绑定的全局帧级图像特征与微表情时间动力学的局部面部特征相融合；（2）话语感知视频 Q-Former，通过为每个话语片段和整个视频生成视觉 token 序列并将其结合，以捕捉多尺度和上下文依赖关系。初步的定性实验表明，在利用多模态和多方面线索以开放词汇（OV）方式预测情感的新型可解释多模态情感识别（EMER）任务中，与最新方法相比，MicroEmo 展现了其有效性。
英文摘要：
Multimodal Large Language Models (MLLMs) have demonstrated remarkable multimodal emotion recognition capabilities, integrating multimodal cues from visual, acoustic, and linguistic contexts in the video to recognize human emotional states. However, existing methods ignore capturing local facial features of temporal dynamics of micro-expressions and do not leverage the contextual dependencies of the utterance-aware temporal segments in the video, thereby limiting their expected effectiveness to a certain extent. In this work, we propose MicroEmo, a time-sensitive MLLM aimed at directing attention to the local facial micro-expression dynamics and the contextual dependencies of utterance-aware video clips. Our model incorporates two key architectural contributions: (1) a global-local attention visual encoder that integrates global frame-level timestamp-bound image features with local facial features of temporal dynamics of micro-expressions; (2) an utterance-aware video Q-Former that captures multi-scale and contextual dependencies by generating visual token sequences for each utterance segment and for the entire video then combining them. Preliminary qualitative experiments demonstrate that in a new Explainable Multimodal Emotion Recognition (EMER) task that exploits multi-modal and multi-faceted clues to predict emotions in an open-vocabulary (OV) manner, MicroEmo demonstrates its effectiveness compared with the latest methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
