本文提出 3D Gaussian Model(3DGM),通过纹理空间绑定与隐式壳层映射,使 3D Gaussian Splatting 支持动画驱动、UV 纹理迁移和交互式编辑。
摘要:
3D Gaussian Splatting 通过实现令人瞩目的保真度和性能,已对神经渲染产生了显著影响。然而,尽管取得了这一成就,它并不容易直接用于开发交互式应用。XR 应用和游戏等实时应用需要动画、UV 映射以及模型编辑等功能,并且这些功能要通过使用一个 3D 模型来同时操控。我们提出一种类似于典型 3D 模型的建模方式,称之为 3D Gaussian Model(3DGM);它为新颖动画和纹理迁移提供了一个可操控的代理。通过在纹理空间中绑定 3D 高斯,并借助隐式壳层映射将其重新投影回世界空间,我们展示了我们的 3D 建模如何能够作为一种适用于交互式应用的有效渲染方法。此外还应注意,近期的 3D 网格重建工作已经能够生成用于渲染的高质量网格。相比之下,我们的工作只需要近似几何,即可高保真地渲染对象。在应用方面,我们将展示基于代理的 3DGM 能够在没有动画训练数据的情况下驱动新颖动画,并能够通过 3D 高斯的 UV 映射进行纹理迁移。我们相信,该结果表明我们的工作具有为 3D Gaussian Splatting 赋能交互式应用的潜力。
英文摘要:
3D Gaussian Splatting has made a marked impact on neural rendering by achieving impressive fidelity and performance. Despite this achievement, however, it is not readily applicable to developing interactive applications. Real-time applications like XR apps and games require functions such as animation, UV-mapping, and model editing simultaneously manipulated through the usage of a 3D model. We propose a modeling that is analogous to typical 3D models, which we call 3D Gaussian Model (3DGM); it provides a manipulatable proxy for novel animation and texture transfer. By binding the 3D Gaussians in texture space and re-projecting them back to world space through implicit shell mapping, we show how our 3D modeling can serve as a valid rendering methodology for interactive applications. It is further noted that recently, 3D mesh reconstruction works have been able to produce high-quality mesh for rendering. Our work, on the other hand, only requires an approximated geometry for rendering an object in high fidelity. Applicationwise, we will show that our proxy-based 3DGM is capable of driving novel animation without animated training data and texture transferring via UV mapping of the 3D Gaussians. We believe the result indicates the potential of our work for enabling interactive applications for 3D Gaussian Splatting.
DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion
The University of Hong Kong (HKU)(香港大学);Astribot(星动科技);Tencent(腾讯);University of Science and Technology of China (USTC)(中国科学技术大学);International Digital Economy Academy (IDEA)(国际数字经济学院)
CAG-Avatar: Cross-Attention Guided Gaussian Avatars for High-Fidelity Head Reconstruction
Dept. of Control Science and Engineering, University of Shanghai for Science and Technology(控制科学与工程系,上海科技大学);School of Psychology and Neuroscience, University of Glasgow(心理学与神经科学学院,格拉斯哥大学)
SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Prometheus Vision Technology Co., Ltd.(天演智视科技有限公司);The Hong Kong University of Science and Technology(香港科技大学)
SEGA: Drivable 3D Gaussian Head Avatar from a Single Image
Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);ByteDance(字节跳动);Pengcheng Lab(鹏城实验室);Department of Automation, Tsinghua University(清华大学自动化系)
Audio-Plane: Audio Factorization Plane Gaussian Splatting for Real-Time Talking Head Synthesis
School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电子与电气工程学院);John A. Paulson School of Engineering and Applied Sciences at Harvard University(哈佛大学约翰·A·保罗森工程与应用科学学院);PhiGent Robotics(PhiGent机器人技术公司);Beijing National Research Center for Information Science and Technology (BNRist) and the Department of Automation, Tsinghua University(北京信息科学与技术国家研究中心(BNRist)和清华大学自动化系)