arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.15708cs.CV
收藏

基于 3D Gaussian Stitching 的真实感基于样例的建模

Towards Realistic Example-based Modeling via 3D Gaussian Stitching

Zhejiang University(浙江大学);The Chinese University of Hong Kong(香港中文大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Xinyu Gao, Ziyi Yang, Bingchen Gong, Xiaoguang Han, Sipeng Yang, Xiaogang Jin

AI总结:

针对真实场景 3D 物体组合难题,提出基于 3DGS 的样例建模方法,利用 GUI 实时分割变换与基于采样的克隆优化,实现无缝且协调的 3D Gaussian 场拼接,显著提升真实感合成效果。

摘要:

利用现有模型的部分来重建新模型,通常被称为基于样例的建模,是计算机图形学领域中的一种经典方法。以往的工作大多侧重于形状组合,这使得它们很难用于对从真实世界场景中捕获的 3D 物体进行真实感的组合。这促使人们将多个 NeRF 结合到一个 3D 场景中,以实现无缝的外观融合。然而,当前的 SeamlessNeRF 方法由于其基于梯度的策略和基于网格的表示,难以在真实世界场景中实现交互式编辑和协调的拼接。为此,我们提出了一种基于样例的建模方法,该方法使用样本引导的合成,在基于点的表示中结合多个 Gaussian 场。具体而言,在组合方面,我们创建了一个 GUI 来实时分割和变换多个场,从而轻松获得由 3D Gaussian Splatting (3DGS) 表示的具有语义意义的模型组合。在纹理混合方面,由于 3DGS 的离散和不规则性质,不支持像 SeamlssNeRF 那样直接应用梯度传播。因此,提出了一种新颖的基于采样的克隆方法,以在保留原始丰富纹理和内容的同时协调混合。我们的工作流程包括三个步骤:1) 使用量身定制的 GUI 对 Gaussian 模型进行实时分割和变换,2) KNN 分析以识别源模型和目标模型之间相交区域的边界点,以及 3) 使用基于采样的克隆和梯度约束对目标模型进行两阶段优化。大量实验结果验证了我们的方法在真实感合成方面显著优于以往工作,展示了其实用性。更多演示可在 https://ingra14m.github.io/gs_stitching_website 获取。

英文摘要:

Using parts of existing models to rebuild new models, commonly termed as example-based modeling, is a classical methodology in the realm of computer graphics. Previous works mostly focus on shape composition, making them very hard to use for realistic composition of 3D objects captured from real-world scenes. This leads to combining multiple NeRFs into a single 3D scene to achieve seamless appearance blending. However, the current SeamlessNeRF method struggles to achieve interactive editing and harmonious stitching for real-world scenes due to its gradient-based strategy and grid-based representation. To this end, we present an example-based modeling method that combines multiple Gaussian fields in a point-based representation using sample-guided synthesis. Specifically, as for composition, we create a GUI to segment and transform multiple fields in real time, easily obtaining a semantically meaningful composition of models represented by 3D Gaussian Splatting (3DGS). For texture blending, due to the discrete and irregular nature of 3DGS, straightforwardly applying gradient propagation as SeamlssNeRF is not supported. Thus, a novel sampling-based cloning method is proposed to harmonize the blending while preserving the original rich texture and content. Our workflow consists of three steps: 1) real-time segmentation and transformation of a Gaussian model using a well-tailored GUI, 2) KNN analysis to identify boundary points in the intersecting area between the source and target models, and 3) two-phase optimization of the target model using sampling-based cloning and gradient constraints. Extensive experimental results validate that our approach significantly outperforms previous works in terms of realistic synthesis, demonstrating its practicality. More demos are available at https://ingra14m.github.io/gs_stitching_website.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑