用于随意拍摄单目视频新视角合成的Dynamic Gaussian Marbles
Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos
AI总结：
针对现有4D高斯方法在随意拍摄单目视频新视角合成中因约束不足效果差的问题，提出Dynamic Gaussian Marbles方法，通过各向同性高斯弹珠、分层分治学习和多尺度先验约束优化，在质量上优于其他高斯基线且保留高斯方法的多项优势。
摘要：
高斯溅射已成为新视角合成的一种流行表征，在效率、光度质量和可编辑性方面展现出明显优势。继其成功之后，许多工作将高斯方法扩展到4D领域，表明动态高斯在保持这些优势的同时，对场景几何的跟踪效果远优于其他替代表征。然而，这些方法都假设以密集多视角视频作为监督信号。在本研究中，我们致力于将高斯场景表征的能力扩展到随意拍摄的单目视频场景中。我们发现，现有的4D高斯方法在该设置下效果极差，因为单目设置存在约束不足的问题。基于这一发现，我们提出了名为Dynamic Gaussian Marbles的方法，它包含三项针对单目场景难点的核心改进。首先，我们采用各向同性的高斯“弹珠”，降低每个高斯体的自由度。其次，我们使用分层分治学习策略，有效引导优化过程向具有全局一致运动的解收敛。最后，我们在优化中加入图像级和几何级先验，包括利用点跟踪领域最新进展的跟踪损失。通过对优化过程施加约束，Dynamic Gaussian Marbles能够学习高斯轨迹，实现新视角渲染，并准确捕捉场景元素的3D运动。我们在Nvidia Dynamic Scenes数据集和DyCheck iPhone数据集上进行了评估，结果表明Gaussian Marbles在质量上显著优于其他高斯基线方法，与非高斯表征效果相当，同时保留了高斯方法的效率、组合性、可编辑性和跟踪优势。我们的项目页面可在此处查看：https://geometry.stanford.edu/projects/dynamic-gaussian-marbles.github.io/。
英文摘要：
Gaussian splatting has become a popular representation for novel-view synthesis, exhibiting clear strengths in efficiency, photometric quality, and compositional edibility. Following its success, many works have extended Gaussians to 4D, showing that dynamic Gaussians maintain these benefits while also tracking scene geometry far better than alternative representations. Yet, these methods assume dense multi-view videos as supervision. In this work, we are interested in extending the capability of Gaussian scene representations to casually captured monocular videos. We show that existing 4D Gaussian methods dramatically fail in this setup because the monocular setting is underconstrained. Building off this finding, we propose a method we call Dynamic Gaussian Marbles, which consist of three core modifications that target the difficulties of the monocular setting. First, we use isotropic Gaussian "marbles'', reducing the degrees of freedom of each Gaussian. Second, we employ a hierarchical divide and-conquer learning strategy to efficiently guide the optimization towards solutions with globally coherent motion. Finally, we add image-level and geometry-level priors into the optimization, including a tracking loss that takes advantage of recent progress in point tracking. By constraining the optimization, Dynamic Gaussian Marbles learns Gaussian trajectories that enable novel-view rendering and accurately capture the 3D motion of the scene elements. We evaluate on the Nvidia Dynamic Scenes dataset and the DyCheck iPhone dataset, and show that Gaussian Marbles significantly outperforms other Gaussian baselines in quality, and is on-par with non-Gaussian representations, all while maintaining the efficiency, compositionality, editability, and tracking benefits of Gaussians. Our project page can be found here https://geometry.stanford.edu/projects/dynamic-gaussian-marbles.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
