面向语音保持的面部表情操控的对比解耦表示学习与正则化
Contrastive Decoupled Representation Learning and Regularization for Speech-Preserving Facial Expression Manipulation
AI总结：
本文提出对比解耦表示学习算法CDRL，通过CCRL和CERL模块分别学习内容与情绪先验并解耦表示，以监督SPFEM生成过程，实现更准确的情绪操控和音唇同步。
摘要：
语音保持的面部表情操控（SPFEM）旨在修改说话人头像，使其显示特定的参考情绪，同时保持源说话内容的口型动画。因此，参考输入和源输入中存在的情绪与内容信息可以为SPFEM模型提供直接且准确的监督信号。然而，这些要素在说话过程中的内在交织对其作为监督信号的有效性构成了挑战。在本工作中，我们提出学习内容和情绪先验作为指导，并通过一种创新的对比解耦表示学习（CDRL）算法结合对比学习来学习解耦的内容和情绪表示。具体而言，设计了一个对比内容表示学习（CCRL）模块来学习主要包含内容信息的音频特征，作为内容先验来指导从源输入中学习内容表示。同时，提出了一个对比情绪表示学习（CERL）模块，利用预训练的视觉-语言模型学习情绪先验，然后用其指导从参考输入中学习情绪表示。我们进一步引入情绪感知和情绪增强的对比学习，分别训练CCRL和CERL模块，确保学习到与情绪无关的内容表示和与内容无关的情绪表示。在SPFEM模型训练期间，解耦的内容和情绪表示被用于监督生成过程，确保更准确的情绪操控以及音频-唇部同步。在各种基准上的大量实验和评估表明了所提算法的有效性。
英文摘要：
Speech-preserving facial expression manipulation (SPFEM) aims to modify a talking head to display a specific reference emotion while preserving the mouth animation of source spoken contents. Thus, emotion and content information existing in reference and source inputs can provide direct and accurate supervision signals for SPFEM models. However, the intrinsic intertwining of these elements during the talking process poses challenges to their effectiveness as supervisory signals. In this work, we propose to learn content and emotion priors as guidance augmented with contrastive learning to learn decoupled content and emotion representation via an innovative Contrastive Decoupled Representation Learning (CDRL) algorithm. Specifically, a Contrastive Content Representation Learning (CCRL) module is designed to learn audio feature, which primarily contains content information, as content priors to guide learning content representation from the source input. Meanwhile, a Contrastive Emotion Representation Learning (CERL) module is proposed to make use of a pre-trained visual-language model to learn emotion prior, which is then used to guide learning emotion representation from the reference input. We further introduce emotion-aware and emotion-augmented contrastive learning to train CCRL and CERL modules, respectively, ensuring learning emotion-independent content representation and content-independent emotion representation. During SPFEM model training, the decoupled content and emotion representations are used to supervise the generation process, ensuring more accurate emotion manipulation together with audio-lip synchronization. Extensive experiments and evaluations on various benchmarks show the effectiveness of the proposed algorithm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
