从静态到动态：适配具有关键点感知能力的图像模型以用于视频中的面部表情识别
From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos
AI总结：
针对视频动态表情识别的数据受限与表情模糊问题，提出S2D模型，将含MCPs的静态ViT通过TMAs扩展到视频，并结合Emotion-Anchors自蒸馏损失，以少量参数增量达到当前最优性能。
摘要：
野外场景下的动态面部表情识别（DFER）仍受到数据限制的阻碍，例如姿态、遮挡和光照方面的数量与多样性不足，以及面部表情固有的模糊性。相比之下，静态面部表情识别（SFER）目前表现出高得多的性能，并且能够受益于更丰富的高质量训练数据。此外，DFER的外观特征和动态依赖关系在很大程度上仍未得到探索。为应对这些挑战，我们提出了一种新颖的静态到动态模型（S2D），该模型利用现有的SFER知识以及隐含编码在所提取的面部关键点感知特征中的动态信息，从而显著提升DFER性能。首先，我们构建并训练一个用于SFER的图像模型，该模型仅包含一个标准Vision Transformer（ViT）和多视角互补提示器（MCPs）。随后，我们通过向图像模型中插入时序建模适配器（TMAs），获得用于DFER的视频模型（即S2D）。MCPs利用由现成面部关键点检测器推断出的关键点感知特征来增强面部表情特征。而TMAs则捕捉并建模面部表情动态变化之间的关系，从而有效地将预训练图像模型扩展到视频。值得注意的是，MCPs和TMAs仅使原始图像模型增加一小部分可训练参数（少于+10%）。此外，我们提出了一种新颖的基于Emotion-Anchors（即每个情绪类别的参考样本）的自蒸馏损失，以减少模糊情绪标签的不利影响，进一步增强我们的S2D。在常用的SFER和DFER数据集上进行的实验表明，我们达到了当前最优水平。
英文摘要：
Dynamic facial expression recognition (DFER) in the wild is still hindered by data limitations, e.g., insufficient quantity and diversity of pose, occlusion and illumination, as well as the inherent ambiguity of facial expressions. In contrast, static facial expression recognition (SFER) currently shows much higher performance and can benefit from more abundant high-quality training data. Moreover, the appearance features and dynamic dependencies of DFER remain largely unexplored. To tackle these challenges, we introduce a novel Static-to-Dynamic model (S2D) that leverages existing SFER knowledge and dynamic information implicitly encoded in extracted facial landmark-aware features, thereby significantly improving DFER performance. Firstly, we build and train an image model for SFER, which incorporates a standard Vision Transformer (ViT) and Multi-View Complementary Prompters (MCPs) only. Then, we obtain our video model (i.e., S2D), for DFER, by inserting Temporal-Modeling Adapters (TMAs) into the image model. MCPs enhance facial expression features with landmark-aware features inferred by an off-the-shelf facial landmark detector. And the TMAs capture and model the relationships of dynamic changes in facial expressions, effectively extending the pre-trained image model for videos. Notably, MCPs and TMAs only increase a fraction of trainable parameters (less than +10\%) to the original image model. Moreover, we present a novel Emotion-Anchors (i.e., reference samples for each emotion category) based Self-Distillation Loss to reduce the detrimental influence of ambiguous emotion labels, further enhancing our S2D. Experiments conducted on popular SFER and DFER datasets show that we achieve the state of the art.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
