模态差异Transformer用于矛盾与犹豫识别

中文摘要 AI 辅助

矛盾与犹豫（A/H）是个体在面部、声音和语言通道中表达相互矛盾信号的情感状态。在临床视频中自动识别A/H需要检测跨模态不一致——这是标准融合方法所抑制的信号。基于Bekhouche等人的冲突感知多模态融合框架，我们提出了模态差异Transformer（MDT）。MDT将原始的6-token设计扩展为9-token表示，包含三个模态嵌入、三个绝对差特征和三个通过线性投影学习的Hadamard积差异特征。这九个token经过Transformer自注意力处理，其中基于FiLM的文本条件调制和LoRA微调是核心架构组件。一个文本引导的晚期融合分支在推理时将仅文本的辅助头与完整的多模态输出相结合。在第三届ABAW挑战赛的BAH数据集上，MDT在标注测试集上达到0.7408的Macro F1，在私有排行榜上达到0.7368，比最强已发表基线高出10多个百分点，同时在单个GPU上训练时间不到20分钟。

英文摘要

Ambivalence and hesitancy (A/H) are affective states in which individuals express contradictory signals across facial, vocal, and linguistic channels. Automatically recognising A/H in clinical videos requires detecting cross-modal disagreement -- the signal that standard fusion methods suppress. Based on the conflict-aware multimodal fusion framework of Bekhouche et al., we present the Modality Discrepancy Transformer (MDT). MDT enriches the original 6-token design to a 9-token representation comprising three modality embeddings, three absolute-difference features, and three Hadamard-product discrepancy features learned through linear projections. These nine tokens undergo Transformer self-attention, with FiLM-based text-conditioned modulation and LoRA fine-tuning as core architectural components. A text-guided late fusion branch blends a text-only auxiliary head with the full multimodal output at inference. On the BAH dataset from the 3rd ABAW Challenge, MDT achieves 0.7408 Macro F1 on the labelled test split and 0.7368 on the private leaderboard, outperforming the strongest published baseline by over 10 points while training in under 20 minutes on a single GPU.

发表机构

• School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
• Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
• School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）
• College of Computer and Information Engineering, Tianjin Normal University（天津师范大学计算机与信息工程学院）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

10 pages