arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.08252cs.CV
收藏

通过因果-扩散桥进行多模态情绪识别(Affect-Diff)

Multimodal Emotion Recognition via Causal-Diffusion Bridge (Affect-Diff)

Department of Computer Science -- Data Science(计算机科学系——数据科学部);Fordham University(福特汉姆大学)

Ankit Sanjyal

AI总结:

针对CMU-MOSEI数据集中情绪类别不平衡问题,Affect-Diff通过因果图、正则化潜在压缩和扩散先验机制提升识别性能,验证了其在平衡准确率和少数类敏感性上的优势。

摘要:

多模态情绪识别在CMU-MOSEI数据集面临极端不平衡问题,因为快乐占样本的65.9%,而三个Ekman类别合计仅占7%。标准融合模型通过忽略少数情绪类别来最大化准确率。我们提出Affect-Diff,一种因果-扩散桥,通过三个联合训练机制解决此问题:一个通过NOTEARS学习的因果图,在融合前重新加权模态贡献;一个beta-VAE瓶颈用于正则化的潜在压缩;以及一个带有stop-gradient的1D DDPM先验,用于对抗多数类崩溃。在3,292个对齐的CMU-MOSEI样本上,Affect-Diff达到验证平衡准确率0.384,比最强基线(TETFN: 0.324)提高了18%。所有评估基线在恐惧、厌恶和惊讶类别上产生零F1。消融研究证实扩散先验和因果图的独立、非冗余贡献。值得注意的是,只有确定性编码器变体能检测所有六个情绪类别,揭示KL正则化强度作为少数类敏感性的直接杠杆。

英文摘要:

Multimodal emotion recognition on CMU-MOSEI faces an extreme imbalance as Happy accounts for 65.9% of samples while three Ekman categories collectively represent under 7%, causing standard fusion models to maximize accuracy by ignoring minority emotions entirely. We present Affect-Diff, a Causal-Diffusion Bridge that addresses this through three jointly trained mechanisms: a NOTEARS-learned causal graph that re-weights modality contributions before fusion, a beta-VAE bottleneck for regularized latent compression, and a stop-gradiented 1D DDPM prior that structures the latent space against majority-class collapse. On 3,292 aligned CMU-MOSEI samples, Affect-Diff achieves validation balanced accuracy 0.384, an 18% relative improvement over the strongest baseline (TETFN: 0.324), while all evaluated baselines produce zero F1 on Fear, Disgust, and Surprise. Ablation studies confirm independent, non-redundant contributions from the diffusion prior (-24% without it) and causal graph (-13%). Notably, only the deterministic-encoder variant detects all six emotion classes, revealing KL regularization strength as a direct lever for minority-class sensitivity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑