CLIP-AUTT: 基于动作单元提示的测试时个性化方法用于细粒度视频情绪识别
CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition
AI总结：
本文提出CLIP-AUTT，通过动作单元提示实现测试时个性化，提升细粒度视频情绪识别的鲁棒性和个性化能力。
摘要：
情绪识别（ER）的个性化对于准确解读细微和特定主体的表达模式至关重要。最近的视觉-语言模型（VLMs）如CLIP展示了在ER中利用联合图像-文本表示的潜力。然而，基于CLIP的方法要么依赖于CLIP的对比预训练，要么依赖LLM生成描述性文本提示，这些提示噪声大、计算成本高且无法捕捉细粒度表达，导致性能下降。本文利用动作单元（AUs）作为结构化文本提示嵌入CLIP中，以建模细粒度面部表情。AUs编码了表达下的细微肌肉激活，提供局部和可解释的语义线索，以实现更稳健的ER。我们引入CLIP-AU，一种轻量级的AU引导时间学习方法，将可解释的AU语义整合到CLIP中。它通过对齐AU提示与面部动态学习通用、主体无关的表示，从而在不进行CLIP微调或LLM生成文本监督的情况下实现细粒度ER。尽管CLIP-AU模型细粒度AU语义，但无法适应主体特定的细微表达变化。为解决这一限制，我们提出CLIP-AUTT，一种基于视频的测试时个性化方法，动态调整AU提示以适应未见过的主体视频。通过结合熵引导的时间窗口选择与提示调优，CLIP-AUTT在保持时间一致性的同时实现主体特定的适应。我们在三个具有挑战性的基于视频的细微ER数据集（BioVid、StressID和BAH）上的广泛实验表明，CLIP-AU和CLIP-AUTT优于最先进的基于CLIP的FER和TTA方法，实现了稳健且个性化的细微ER。我们的代码已公开在：https://github.com/osamazeeshan/CLIP-AUTT。
英文摘要：
Personalization in emotion recognition (ER) is essential for accurate interpretation of subtle and subject-specific expressive patterns. Recent advances in vision-language models (VLMs), such as CLIP, demonstrate strong potential for leveraging joint image-text representations in ER. However, existing CLIP-based methods either rely on CLIP's contrastive pretraining or use LLMs to generate descriptive text prompts, which can be noisy, computationally expensive, and often fail to capture fine-grained expressions, leading to degraded performance. In this work, Action Units (AUs) are leveraged as structured textual prompts within CLIP to model fine-grained facial expressions. AUs encode the subtle muscle activations underlying expressions, providing localized and interpretable semantic cues for more robust facial expression recognition (FER). We introduce CLIP-AU, a lightweight AU-guided temporal learning method that integrates interpretable AU semantics into CLIP. It learns generic, subject-agnostic representations by aligning AU prompts with facial dynamics, enabling fine-grained FER without CLIP fine-tuning or LLM-generated text supervision. Although CLIP-AU models fine-grained AU semantics, it does not adapt to subject-specific variability in subtle expressions. To address this limitation, we propose CLIP-AUTT, a video-based test-time personalization method that dynamically adapts AU prompts to videos from unseen subjects. By combining entropy-guided temporal window selection with prompt tuning, CLIP-AUTT enables subject-specific adaptation while preserving temporal consistency. Our experiments on three challenging video-based datasets, BioVid, StressID, and BAH, indicate that CLIP-AU and CLIP-AUTT outperform state-of-the-art CLIP-based FER and TTA methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
