MuMTAffect：一种用于从生理信号中识别人格与情绪的多模态多任务情感框架
MuMTAffect: A Multimodal Multitask Affective Framework for Personality and Emotion Recognition from Physiological Signals
AI总结：
本研究提出多模态多任务情感框架MuMTAffect，结合多种生理信号与transformer架构，以人格预测为辅助任务提升情绪识别性能，在AFFEC数据集上验证了有效性与可扩展性。
摘要：
我们提出了MuMTAffect，这是一种新型的多模态多任务情感嵌入网络，旨在从短生理信号片段中联合完成情绪分类和人格预测（重识别）。MuMTAffect整合了多种生理模态：瞳孔扩张、眼动追踪、面部动作单元和皮肤电反应，为每种模态配备专用的基于transformer的编码器，并通过一个融合transformer建模跨模态交互。受情绪建构理论启发，该架构明确将核心情感编码（效价/唤醒度）与更高层级的概念化过程分离，从而让预测建立在当代情感神经科学的基础之上。人格特质预测被用作辅助任务，以生成鲁棒的、用户特定的情感嵌入，显著提升情绪识别性能。我们在AFFEC数据集上对MuMTAffect进行了评估，结果表明刺激级情绪线索（Stim Emo）和皮肤电反应大幅提升了唤醒度分类效果，而瞳孔和眼动数据则增强了效价区分能力。MuMTAffect固有的模块化特性使得新增模态的集成十分简便，确保了可扩展性和适应性。大量实验和消融研究证实了我们的多模态多任务方法在构建个性化、情境感知的情感计算系统方面的有效性，为跨被试泛化能力的进一步提升指明了方向。
英文摘要：
We present MuMTAffect, a novel Multimodal Multitask Affective Embedding Network designed for joint emotion classification and personality prediction (re-identification) from short physiological signal segments. MuMTAffect integrates multiple physiological modalities pupil dilation, eye gaze, facial action units, and galvanic skin response using dedicated, transformer-based encoders for each modality and a fusion transformer to model cross-modal interactions. Inspired by the Theory of Constructed Emotion, the architecture explicitly separates core affect encoding (valence/arousal) from higher-level conceptualization, thereby grounding predictions in contemporary affective neuroscience. Personality trait prediction is leveraged as an auxiliary task to generate robust, user-specific affective embeddings, significantly enhancing emotion recognition performance. We evaluate MuMTAffect on the AFFEC dataset, demonstrating that stimulus-level emotional cues (Stim Emo) and galvanic skin response substantially improve arousal classification, while pupil and gaze data enhance valence discrimination. The inherent modularity of MuMTAffect allows effortless integration of additional modalities, ensuring scalability and adaptability. Extensive experiments and ablation studies underscore the efficacy of our multimodal multitask approach in creating personalized, context-aware affective computing systems, highlighting pathways for further advancements in cross-subject generalisation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
