面向距离的软提示学习用于多模态愉悦-唤醒估计
Distance-aware Soft Prompt Guidance for Multimodal Valence-Arousal Estimation
AI总结：
本文提出一种多模态框架，通过引入距离感知的软提示学习，提升多模态愉悦-唤醒估计的精度，在无约束场景中取得良好效果。
摘要：
愉悦-唤醒（VA）估计对于捕捉自然环境中人类情感的细微特性至关重要。尽管预训练的视觉-语言模型如CLIP展示了显著的语义对齐能力，但其在连续回归任务中的应用常受限于文本提示的离散性质。本文提出了一种新颖的多模态框架，引入距离感知的软提示学习以弥合语义空间与连续维度之间的差距。具体而言，我们将VA空间划分为3×3网格，定义九个情感区域，每个区域关联有独特的文本描述。不同于硬分类，我们采用高斯核计算基于真实坐标与区域中心之间欧几里得距离的软标签，使模型能够学习细腻的情感过渡。对于多模态整合，我们的架构利用CLIP图像编码器和音频频谱变压器（AST）提取稳健的空间和声学特征。这些特征通过门控循环单元（GRUs）进行时间建模，并通过层次融合方案整合，该方案依次结合跨模态注意力进行对齐和门控融合进行自适应细化。在Aff-Wild2数据集上的实验结果表明，我们提出的语义引导方法显著提升了VA估计的准确性，在无约束的``in-the-wild''场景中取得了竞争性的性能。
英文摘要：
Valence-arousal (VA) estimation is crucial for capturing the nuanced nature of human emotions in naturalistic environments. While pre-trained vision-language models such as CLIP have demonstrated remarkable semantic alignment capabilities, their application to continuous regression tasks is often limited by the discrete nature of text prompts. In this paper, we propose a novel multimodal framework for VA estimation that introduces Distance-aware Soft Prompt Guidance to bridge the gap between semantic representations and continuous affective dimensions. Specifically, we partition the VA space into multiple discrete regions, each associated with distinct textual descriptions. Rather than relying on hard categorization, we employ a Gaussian kernel to compute soft labels based on the Euclidean distance between the ground-truth coordinates and the region centers, allowing the model to learn fine-grained emotional transitions. For multimodal integration, our architecture utilizes a CLIP image encoder and an Audio Spectrogram Transformer to extract robust visual and acoustic features. These features are temporally modeled using Gated Recurrent Units and integrated through a hierarchical fusion scheme that sequentially combines cross-modal attention for alignment and gated fusion for adaptive refinement. Experimental results on the Aff-Wild2 dataset show that the proposed semantic-guided approach outperforms the official baseline and demonstrates robust performance on in-the-wild data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
