arXiv 2402.061492024-02cs.CV

HeadStudio：基于3D Gaussian Splatting的文本到可动画头部虚拟人

HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splatting

Zhejiang University（浙江大学）；ReLER, CCAI, Zhejiang University（浙江大学ReLER、CCAI）；State Key Laboratory of Brain-machine Intelligence, Zhejiang University（浙江大学脑机智能国家重点实验室）

Zhenglin Zhou, Fan Ma, Hehe Fan, Zongxin Yang, Yi Yang

备注：: 26 pages, 18 figures, accepted by ECCV 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出HeadStudio框架，利用3D Gaussian Splatting结合可动画头部先验模型，通过增强初始化、蒸馏和正则化优化，实现从文本生成高质量、可实时渲染且能由语音和视频驱动的可动画头部虚拟人。

摘要：

从文本提示创建数字虚拟人一直是一项令人向往且充满挑战的任务。尽管2D扩散先验取得了有前景的结果，但当前方法仍难以高效创建高质量且一致的可动画虚拟人。以往的可动画头部模型（如FLAME）难以准确表示详细的纹理和几何结构。此外，高质量的3D静态表示在利用动态先验进行语义驱动方面也面临挑战。在本文中，我们提出了HeadStudio，这是一个利用3D Gaussian Splatting从文本提示生成逼真且可动画虚拟人的新框架。首先，我们将3D Gaussians与可动画头部先验模型相关联，促进了在高质量3D表示上的语义动画。为确保动画的一致性，我们进一步从初始化、蒸馏和正则化方面增强优化过程，以联合学习形状、纹理和动画。大量实验证明了HeadStudio在从文本提示生成可动画虚拟人方面的有效性，展现出吸引人的外观。这些虚拟人能够在1024分辨率下渲染高质量的实时（≥40 fps）新视角。此外，这些虚拟人可以由真实世界的语音和视频平滑驱动。我们希望HeadStudio能推动数字虚拟人的创建并在社区中获得普及。代码位于：https://github.com/ZhenglinZhou/HeadStudio。

英文摘要：

Creating digital avatars from textual prompts has long been a desirable yet challenging task. Despite the promising results achieved with 2D diffusion priors, current methods struggle to create high-quality and consistent animated avatars efficiently. Previous animatable head models like FLAME have difficulty in accurately representing detailed texture and geometry. Additionally, high-quality 3D static representations face challenges in semantically driving with dynamic priors. In this paper, we introduce \textbf{HeadStudio}, a novel framework that utilizes 3D Gaussian splatting to generate realistic and animatable avatars from text prompts. Firstly, we associate 3D Gaussians with animatable head prior model, facilitating semantic animation on high-quality 3D representations. To ensure consistent animation, we further enhance the optimization from initialization, distillation, and regularization to jointly learn the shape, texture, and animation. Extensive experiments demonstrate the efficacy of HeadStudio in generating animatable avatars from textual prompts, exhibiting appealing appearances. The avatars are capable of rendering high-quality real-time ($\geq 40$ fps) novel views at a resolution of 1024. Moreover, These avatars can be smoothly driven by real-world speech and video. We hope that HeadStudio can enhance digital avatar creation and gain popularity in the community. Code is at: https://github.com/ZhenglinZhou/HeadStudio.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文