DaGAN++：用于说话头部视频生成的深度感知生成对抗网络
DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video Generation
AI总结：
本文提出 DaGAN++，通过自监督深度学习、几何引导关键点估计和三维感知跨模态注意力生成逼真说话头部视频，并在 VoxCeleb1、VoxCeleb2、HDTF 上达到最先进水平。
摘要：
说话头部生成的主流技术在很大程度上依赖于二维信息，包括来自输入人脸图像的面部外观和运动。然而，诸如逐像素深度之类的密集三维面部几何，在构建准确的三维面部结构并抑制生成过程中复杂背景噪声方面起着关键作用。但是，获取面部视频的密集三维标注成本高得令人却步。在本工作中，首先，我们提出了一种新颖的自监督方法，用于从面部视频中学习密集三维面部几何（即深度），训练时不需要相机参数和三维几何标注。我们进一步提出了一种学习像素级不确定性的策略，以感知更可靠的刚性运动像素，用于几何学习。其次，我们设计了一个有效的几何引导面部关键点估计模块，为生成运动场提供准确的关键点。最后，我们开发了一种三维感知的跨模态（即外观和深度）注意力机制，该机制可应用于每个生成层，以由粗到细的方式捕捉面部几何。我们在三个具有挑战性的基准（即 VoxCeleb1、VoxCeleb2 和 HDTF）上进行了大量实验。结果表明，我们提出的框架能够生成高度逼真的重演说话视频，并在这些基准上确立了新的最先进性能。代码和训练模型已公开发布在 GitHub 项目页面：https://github.com/harlanhong/CVPR2022-DaGAN
英文摘要：
Predominant techniques on talking head generation largely depend on 2D information, including facial appearances and motions from input face images. Nevertheless, dense 3D facial geometry, such as pixel-wise depth, plays a critical role in constructing accurate 3D facial structures and suppressing complex background noises for generation. However, dense 3D annotations for facial videos is prohibitively costly to obtain. In this work, firstly, we present a novel self-supervised method for learning dense 3D facial geometry (ie, depth) from face videos, without requiring camera parameters and 3D geometry annotations in training. We further propose a strategy to learn pixel-level uncertainties to perceive more reliable rigid-motion pixels for geometry learning. Secondly, we design an effective geometry-guided facial keypoint estimation module, providing accurate keypoints for generating motion fields. Lastly, we develop a 3D-aware cross-modal (ie, appearance and depth) attention mechanism, which can be applied to each generation layer, to capture facial geometries in a coarse-to-fine manner. Extensive experiments are conducted on three challenging benchmarks (ie, VoxCeleb1, VoxCeleb2, and HDTF). The results demonstrate that our proposed framework can generate highly realistic-looking reenacted talking videos, with new state-of-the-art performances established on these benchmarks. The codes and trained models are publicly available on the GitHub project page at https://github.com/harlanhong/CVPR2022-DaGAN
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
