GenCA：一种用于逼真且可驱动的编解码器虚拟人的文本条件生成模型
GenCA: A Text-conditioned Generative Model for Realistic and Drivable Codec Avatars
AI总结：
提出文本条件生成模型GenCA，结合潜在扩散模型与表情驱动先验，生成可驱动的照片级逼真3D虚拟人，支持虚拟人编辑与单样本重建。
摘要：
照片级逼真且可控的3D虚拟人对虚拟和混合现实（VR/MR）、远程呈现、游戏和电影制作等各种应用至关重要。传统的虚拟人创建方法通常涉及针对每个虚拟人的耗时扫描和重建过程，这限制了其可扩展性。此外，这些方法无法灵活地采样新身份或修改现有身份。另一方面，通过从数据中学习强先验，生成模型为传统重建方法提供了一种有前景的替代方案，减轻了数据采集和处理的时间限制。此外，生成方法支持重建之外的下游应用，如编辑和风格化。尽管如此，生成式3D虚拟人的研究仍处于起步阶段，因此当前方法仍存在局限性，例如只能创建静态虚拟人、缺乏照片级真实感、面部细节不完整或可驱动性有限。为了解决这些问题，我们提出了一种文本条件生成模型，能够生成具有多样身份的照片级逼真面部虚拟人，包含更完整的细节（如头发、眼睛和口腔内部），并可通过强大的非参数化潜在表情空间进行驱动。具体而言，我们将潜在扩散模型的生成和编辑能力与用于虚拟人表情驱动的强先验模型相结合。我们的模型能够生成和控制高保真虚拟人，甚至是分布外的虚拟人。我们还强调了其在下游应用中的潜力，包括虚拟人编辑和单样本虚拟人重建。
英文摘要：
Photo-realistic and controllable 3D avatars are crucial for various applications such as virtual and mixed reality (VR/MR), telepresence, gaming, and film production. Traditional methods for avatar creation often involve time-consuming scanning and reconstruction processes for each avatar, which limits their scalability. Furthermore, these methods do not offer the flexibility to sample new identities or modify existing ones. On the other hand, by learning a strong prior from data, generative models provide a promising alternative to traditional reconstruction methods, easing the time constraints for both data capture and processing. Additionally, generative methods enable downstream applications beyond reconstruction, such as editing and stylization. Nonetheless, the research on generative 3D avatars is still in its infancy, and therefore current methods still have limitations such as creating static avatars, lacking photo-realism, having incomplete facial details, or having limited drivability. To address this, we propose a text-conditioned generative model that can generate photo-realistic facial avatars of diverse identities, with more complete details like hair, eyes and mouth interior, and which can be driven through a powerful non-parametric latent expression space. Specifically, we integrate the generative and editing capabilities of latent diffusion models with a strong prior model for avatar expression driving. Our model can generate and control high-fidelity avatars, even those out-of-distribution. We also highlight its potential for downstream applications, including avatar editing and single-shot avatar reconstruction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
