基于视觉原型的情感识别基础：VEGA——重新审视CLIP在MERC中的应用
Grounding Emotion Recognition with Visual Prototypes: VEGA -- Revisiting CLIP in MERC
AI总结：
本文提出视觉情感引导锚定（VEGA）机制，利用CLIP图像编码器构建情感视觉锚点，结合随机锚点采样与自蒸馏双分支架构，在IEMOCAP和MELD基准上实现SOTA多模态情感识别性能。
摘要：
由于文本、声学和视觉信号之间复杂的相互作用，对话中的多模态情感识别仍然是一项具有挑战性的任务。尽管最近的模型通过高级融合策略提高了性能，但它们通常缺乏心理学意义的先验来指导多模态对齐。在本文中，我们重新审视了CLIP的使用，并提出了一种新颖的视觉情感引导锚定（VEGA）机制，该机制将类级别的视觉语义引入融合和分类过程。与主要利用CLIP文本编码器的先前工作不同，我们的方法利用其图像编码器基于面部样本构建特定情感的视觉锚点。受认知理论（原型情感类别和多感觉整合）启发，这些锚点将单模态和多模态特征引导至感知基础且心理对齐的表示空间。随机锚点采样策略通过平衡语义稳定性和类内多样性进一步增强了鲁棒性。我们的VEGA增强模型集成到具有自蒸馏的双分支架构中，在IEMOCAP和MELD上实现了sota性能。代码可在：https://github.com/dkollias/VEGA 获取。
英文摘要：
Multimodal Emotion Recognition in Conversations remains a challenging task due to the complex interplay of textual, acoustic and visual signals. While recent models have improved performance via advanced fusion strategies, they often lack psychologically meaningful priors to guide multimodal alignment. In this paper, we revisit the use of CLIP and propose a novel Visual Emotion Guided Anchoring (VEGA) mechanism that introduces class-level visual semantics into the fusion and classification process. Distinct from prior work that primarily utilizes CLIP's textual encoder, our approach leverages its image encoder to construct emotion-specific visual anchors based on facial exemplars. These anchors guide unimodal and multimodal features toward a perceptually grounded and psychologically aligned representation space, drawing inspiration from cognitive theories (prototypical emotion categories and multisensory integration). A stochastic anchor sampling strategy further enhances robustness by balancing semantic stability and intra-class diversity. Integrated into a dual-branch architecture with self-distillation, our VEGA-augmented model achieves sota performance on IEMOCAP and MELD. Code is available at: https://github.com/dkollias/VEGA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
