A$^{3}$lign-DFER：面向基于CLIP的动态面部表情识别的开创性综合动态情感对齐方法
A$^{3}$lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP
AI总结：
针对CLIP在动态面部表情识别任务中对齐困难、表现不佳的问题，提出A$^{3}$lign-DFER方法，通过多维对齐标记、联合动态对齐同步器和双向对齐训练范式实现多维度对齐，在多个DFER数据集上取得当前最优结果。
摘要：
CLIP在动态面部表情识别（DFER）任务中的表现并未像在其他基于CLIP的分类任务中那样取得优异成果。CLIP的核心目标是在特征空间中实现图像与文本的对齐，但DFER任务因文本的抽象性和视频的动态性带来挑战，导致标签表征受限，难以实现完美对齐。为解决这一问题，我们设计了A$^{3}$lign-DFER，它引入了一种全新的DFER标注范式以全面实现对齐，从而提升CLIP对DFER任务的适配性。具体而言，我们的A$^{3}$lign-DFER方法包含多个协同工作的模块，以获取最适合分类的维度扩展嵌入，并在情感、动态和双向三个关键层面实现对齐。我们用可学习的多维对齐标记（Multi-Dimensional Alignment Token，MAT）替换输入的标签文本，使文本能够在情感和动态两个维度上与面部表情视频样本对齐。在CLIP特征提取之后，我们引入联合动态对齐同步器（Joint Dynamic Alignment Synchronizer，JAS），进一步促进时间维度上的同步与对齐。此外，我们还实现了双向对齐训练范式（Bidirectional Alignment Training Paradigm，BAP），以确保两种模态的参数能够逐步稳定地训练。我们兼具创新性与简洁性的A$^{3}$lign-DFER方法在DFEW、FERV39k和MAFW等多个DFER数据集上取得了当前最优（state-of-the-art）结果。大量消融实验和可视化研究验证了A$^{3}$lign-DFER的有效性。代码将于未来公开。
英文摘要：
The performance of CLIP in dynamic facial expression recognition (DFER) task doesn't yield exceptional results as observed in other CLIP-based classification tasks. While CLIP's primary objective is to achieve alignment between images and text in the feature space, DFER poses challenges due to the abstract nature of text and the dynamic nature of video, making label representation limited and perfect alignment difficult. To address this issue, we have designed A$^{3}$lign-DFER, which introduces a new DFER labeling paradigm to comprehensively achieve alignment, thus enhancing CLIP's suitability for the DFER task. Specifically, our A$^{3}$lign-DFER method is designed with multiple modules that work together to obtain the most suitable expanded-dimensional embeddings for classification and to achieve alignment in three key aspects: affective, dynamic, and bidirectional. We replace the input label text with a learnable Multi-Dimensional Alignment Token (MAT), enabling alignment of text to facial expression video samples in both affective and dynamic dimensions. After CLIP feature extraction, we introduce the Joint Dynamic Alignment Synchronizer (JAS), further facilitating synchronization and alignment in the temporal dimension. Additionally, we implement a Bidirectional Alignment Training Paradigm (BAP) to ensure gradual and steady training of parameters for both modalities. Our insightful and concise A$^{3}$lign-DFER method achieves state-of-the-art results on multiple DFER datasets, including DFEW, FERV39k, and MAFW. Extensive ablation experiments and visualization studies demonstrate the effectiveness of A$^{3}$lign-DFER. The code will be available in the future.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
