CLAIP-Emo：面向自然场景音视频情感识别的语言监督模型参数高效适配
CLAIP-Emo: Parameter-Efficient Adaptation of Language-supervised models for In-the-Wild Audiovisual Emotion Recognition
AI总结：
本文提出CLAIP-Emo框架，通过冻结CLIP/CLAP骨干并用LoRA进行参数高效适配，结合非对称时序建模和简单融合头，在DFEW和MAFW上以仅8M训练参数取得了音视频情感识别的新最优性能。
摘要：
自然场景下的音视频情感识别（AVER）仍然受到姿态变化、遮挡和背景噪声的阻碍。现有方法主要依赖大规模领域特定预训练，这不仅成本高昂，而且常常与真实世界的情感数据不匹配。为了解决这一问题，我们提出了CLAIP-Emo，这是一个模块化框架，将自然场景AVER重新构建为语言监督基础模型（CLIP/CLAP）的参数高效适配。具体而言，该框架（i）通过冻结CLIP/CLAP骨干网络并利用LoRA进行面向情感的适配（更新总参数的≤4.0%）来保留语言监督先验，（ii）非对称地分配时序建模，采用轻量级Transformer处理视觉动态，同时对音频韵律应用均值池化，（iii）应用一个简单的融合头进行预测。在DFEW和MAFW数据集上，CLAIP-Emo（ViT-L/14）仅使用8M训练参数就分别达到了80.14%和61.18%的加权平均召回率，创造了新的最优性能。我们的研究结果表明，语言监督基础模型的参数高效适配为真实世界AVER提供了一种可扩展的、替代领域特定预训练的方法。代码和模型将在https://github.com/MSA-LMC/CLAIP-Emo上公开。
英文摘要：
Audiovisual emotion recognition (AVER) in the wild is still hindered by pose variation, occlusion, and background noise. Prevailing methods primarily rely on large-scale domain-specific pre-training, which is costly and often mismatched to real-world affective data. To address this, we present CLAIP-Emo, a modular framework that reframes in-the-wild AVER as a parameter-efficient adaptation of language-supervised foundation models (CLIP/CLAP). Specifically, it (i) preserves language-supervised priors by freezing CLIP/CLAP backbones and performing emotion-oriented adaptation via LoRA (updating \ensuremath{\le}4.0\% of the total parameters), (ii) allocates temporal modeling asymmetrically, employing a lightweight Transformer for visual dynamics while applying mean pooling for audio prosody, and (iii) applies a simple fusion head for prediction. On DFEW and MAFW, CLAIP-Emo (ViT-L/14) achieves 80.14\% and 61.18\% weighted average recall with only 8M training parameters, setting a new state of the art. Our findings suggest that parameter-efficient adaptation of language-supervised foundation models provides a scalable alternative to domain-specific pre-training for real-world AVER. The code and models will be available at \href{https://github.com/MSA-LMC/CLAIP-Emo}{https://github.com/MSA-LMC/CLAIP-Emo}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
