情境融合：一种面向情感状态识别的多模态方法
Fusion in Context: A Multimodal Approach to Affective State Recognition
AI总结：
针对情绪表达受情境影响易误判的问题，提出基于transformer的多模态融合方法，结合面部热数据、动作单元与文本情境，通过模态专属编码器、加法融合与共享transformer实现情境感知情绪识别，在桌面Pacman游戏数据集上验证了有效性。
摘要：
准确识别人类情绪是情感计算与人机交互（HRI）领域的关键挑战。情绪状态在塑造行为、决策与社会互动方面发挥着重要作用。然而，情绪表达会受到情境因素的影响，若不考虑情境则会导致解读偏差。融合面部表情、语音、生理信号等模态的多模态融合技术，已在提升情绪识别效果方面展现出潜力。本文提出一种基于transformer的多模态融合方法，利用面部热成像数据、面部动作单元与文本情境信息，实现具备情境感知能力的情绪识别。我们采用模态专属编码器学习定制化表征，再通过加法融合对表征进行融合，随后由共享transformer编码器处理以捕捉时序依赖与模态间交互。研究在一个数据集上评估了所提方法，该数据集采集自参与实体桌面Pacman游戏的受试者，游戏设计旨在诱发多种情绪状态。结果证明，引入情境信息与多模态融合对情感状态识别具有有效性。
英文摘要：
Accurate recognition of human emotions is a crucial challenge in affective computing and human-robot interaction (HRI). Emotional states play a vital role in shaping behaviors, decisions, and social interactions. However, emotional expressions can be influenced by contextual factors, leading to misinterpretations if context is not considered. Multimodal fusion, combining modalities like facial expressions, speech, and physiological signals, has shown promise in improving affect recognition. This paper proposes a transformer-based multimodal fusion approach that leverages facial thermal data, facial action units, and textual context information for context-aware emotion recognition. We explore modality-specific encoders to learn tailored representations, which are then fused using additive fusion and processed by a shared transformer encoder to capture temporal dependencies and interactions. The proposed method is evaluated on a dataset collected from participants engaged in a tangible tabletop Pacman game designed to induce various affective states. Our results demonstrate the effectiveness of incorporating contextual information and multimodal fusion for affective state recognition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
