PV3D：用于肖像视频生成的3D生成模型
PV3D: A 3D Generative Model for Portrait Video Generation
AI总结：
PV3D是首个能生成多视角一致肖像视频的3D生成框架，通过扩展3D隐式神经表示到时空域，结合运动生成器和相机条件策略，实现高质量、运动合理的视频生成，并支持多种下游应用。
摘要：
生成对抗网络（GANs）的最新进展已展现出生成令人惊叹的逼真肖像图像的能力。虽然一些先前的工作已将此类图像GAN应用于无条件的2D肖像视频生成和静态3D肖像合成，但很少有工作成功地将GAN扩展用于生成3D感知的肖像视频。在这项工作中，我们提出了PV3D，这是第一个能够合成多视角一致肖像视频的生成框架。具体来说，我们的方法通过将3D隐式神经表示推广到时空空间建模，将最近的静态3D感知图像GAN扩展到视频领域。为了在生成过程中引入运动动态，我们开发了一个运动生成器，通过堆叠多个运动层，利用调制卷积生成运动特征。为了减轻由相机/人体运动引起的运动模糊性，我们为PV3D提出了一种简单而有效的相机条件策略，实现了时间一致和多视角一致的视频生成。此外，PV3D引入了两个判别器，分别用于规范空间和时间域，以确保生成的肖像视频的合理性。这些精心设计使PV3D能够生成具有高质量外观和几何形状的3D感知、运动合理的肖像视频，显著优于先前的工作。因此，PV3D能够支持许多下游应用，如动画化静态肖像和视图一致的视频运动编辑。代码和模型已在https://showlab.github.io/pv3d发布。
英文摘要：
Recent advances in generative adversarial networks (GANs) have demonstrated the capabilities of generating stunning photo-realistic portrait images. While some prior works have applied such image GANs to unconditional 2D portrait video generation and static 3D portrait synthesis, there are few works successfully extending GANs for generating 3D-aware portrait videos. In this work, we propose PV3D, the first generative framework that can synthesize multi-view consistent portrait videos. Specifically, our method extends the recent static 3D-aware image GAN to the video domain by generalizing the 3D implicit neural representation to model the spatio-temporal space. To introduce motion dynamics to the generation process, we develop a motion generator by stacking multiple motion layers to generate motion features via modulated convolution. To alleviate motion ambiguities caused by camera/human motions, we propose a simple yet effective camera condition strategy for PV3D, enabling both temporal and multi-view consistent video generation. Moreover, PV3D introduces two discriminators for regularizing the spatial and temporal domains to ensure the plausibility of the generated portrait videos. These elaborated designs enable PV3D to generate 3D-aware motion-plausible portrait videos with high-quality appearance and geometry, significantly outperforming prior works. As a result, PV3D is able to support many downstream applications such as animating static portraits and view-consistent video motion editing. Code and models are released at https://showlab.github.io/pv3d.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
