arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.19035cs.CV
收藏

ScalingGaussian:借助生成式高斯泼溅增强3D内容创作

ScalingGaussian: Enhancing 3D Content Creation with Generative Gaussian Splatting

School of Information Science and Engineering, East China University of Science and Technology(华东理工大学信息科学与工程学院);University of Washington(华盛顿大学);University of Copenhagen(哥本哈根大学);Tsinghua University(清华大学);FancyTech(范希科技(FancyTech))

Shen Chen, Jiale Zhou, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, Lei Li

AI总结:

提出ScalingGaussian框架,结合3D和2D扩散模型,通过点云致密化、SDS损失细化高斯及MSE与GPP损失优化纹理,解决点云稀疏问题,实现高质量3D资产生成。

摘要:

高质量3D资产生成对于数字遗产保护、娱乐和机器人应用至关重要。传统上,这一过程需要熟练的专业人员和专业软件来对3D对象进行建模、纹理处理和渲染。然而,游戏和虚拟现实(VR)领域对3D资产日益增长的需求促使了易于使用的图像到3D技术的出现,使得非专业人员能够制作3D内容并减少对专家的依赖。现有的3D内容生成方法难以同时实现精细的纹理和强大的几何一致性。我们提出了一种新颖的3D内容创作框架ScalingGaussian,它结合了3D和2D扩散模型,以在生成的3D资产中实现精细的纹理和几何一致性。首先,3D扩散模型生成点云,随后通过选择局部区域、引入高斯噪声并使用局部密度加权选择的过程对其进行致密化。为了细化3D高斯,我们利用带有Score Distillation Sampling (SDS)损失的2D扩散模型,引导3D高斯进行克隆和分裂。最后,将3D高斯转换为网格,并使用Mean Square Error (MSE)和Gradient Profile Prior (GPP)损失对表面纹理进行优化。我们的方法解决了3D扩散中常见的点云稀疏问题,从而改善了几何结构和纹理细节。在图像到3D任务上的实验表明,我们的方法能够高效生成高质量的3D资产。

英文摘要:

The creation of high-quality 3D assets is paramount for applications in digital heritage preservation, entertainment, and robotics. Traditionally, this process necessitates skilled professionals and specialized software for the modeling, texturing, and rendering of 3D objects. However, the rising demand for 3D assets in gaming and virtual reality (VR) has led to the creation of accessible image-to-3D technologies, allowing non-professionals to produce 3D content and decreasing dependence on expert input. Existing methods for 3D content generation struggle to simultaneously achieve detailed textures and strong geometric consistency. We introduce a novel 3D content creation framework, ScalingGaussian, which combines 3D and 2D diffusion models to achieve detailed textures and geometric consistency in generated 3D assets. Initially, a 3D diffusion model generates point clouds, which are then densified through a process of selecting local regions, introducing Gaussian noise, followed by using local density-weighted selection. To refine the 3D gaussians, we utilize a 2D diffusion model with Score Distillation Sampling (SDS) loss, guiding the 3D Gaussians to clone and split. Finally, the 3D Gaussians are converted into meshes, and the surface textures are optimized using Mean Square Error(MSE) and Gradient Profile Prior(GPP) losses. Our method addresses the common issue of sparse point clouds in 3D diffusion, resulting in improved geometric structure and detailed textures. Experiments on image-to-3D tasks demonstrate that our approach efficiently generates high-quality 3D assets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑