高保真且可自由控制的说话头部视频生成
High-Fidelity and Freely Controllable Talking Head Video Generation
AI总结：
针对说话头部视频中的人脸失真、属性控制不足和帧间闪烁问题，提出融合自监督与3D人脸关键点的模型，并结合运动感知对齐及上下文传播模块，实现高保真、姿态与表情可控的生成。
摘要：
说话头部生成旨在根据给定的源身份和目标运动生成视频。然而，当前方法面临若干挑战，限制了生成视频的质量和可控性。首先，生成的人脸常出现意外变形和严重失真。其次，驱动图像没有显式解耦与运动相关的信息，例如姿态和表情，这限制了生成过程中对不同属性的操控。第三，由于相邻帧之间提取的关键点不一致，生成的视频往往存在闪烁伪影。本文提出一种新颖模型，可生成高保真说话头部视频，并能自由控制头部姿态和表情。我们的方法同时利用自监督学习得到的关键点和基于3D人脸模型的关键点来建模运动。我们还引入了新颖的运动感知多尺度特征对齐模块，以有效迁移运动且不造成人脸失真。此外，我们通过特征上下文适应与传播模块增强合成说话头部视频的平滑性。我们在具有挑战性的数据集上评估了该模型，并证明其达到最先进性能。
英文摘要：
Talking head generation is to generate video based on a given source identity and target motion. However, current methods face several challenges that limit the quality and controllability of the generated videos. First, the generated face often has unexpected deformation and severe distortions. Second, the driving image does not explicitly disentangle movement-relevant information, such as poses and expressions, which restricts the manipulation of different attributes during generation. Third, the generated videos tend to have flickering artifacts due to the inconsistency of the extracted landmarks between adjacent frames. In this paper, we propose a novel model that produces high-fidelity talking head videos with free control over head pose and expression. Our method leverages both self-supervised learned landmarks and 3D face model-based landmarks to model the motion. We also introduce a novel motion-aware multi-scale feature alignment module to effectively transfer the motion without face distortion. Furthermore, we enhance the smoothness of the synthesized talking head videos with a feature context adaptation and propagation module. We evaluate our model on challenging datasets and demonstrate its state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
