arXiv 2412.024932024-12-03cs.CV

RelayGS：通过接力高斯重建具有大规模复杂运动的动态场景

RelayGS: Reconstructing Dynamic Scenes with Large-Scale and Complex Motions via Relay Gaussians

School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology（广东省超高清沉浸式媒体技术重点实验室）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）

Qiankun Gao, Yanmin Wu, Chengxiang Wen, Jiarui Meng, Luyang Tang, Jie Chen, Ronggang Wang, Jian Zhang

备注：: Technical Report. GitHub: https://github.com/gqk/RelayGS

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对3DGS难以处理大幅复杂运动的问题，本文提出三阶段RelayGS，通过接力高斯分解运动轨迹并联合优化4D表示，PSNR提升超1 dB。

摘要：

重建具有大规模复杂运动的动态场景仍然是一项重大挑战。神经辐射场和3D Gaussian Splatting（3DGS）等近期技术已展现出良好前景，但在涉及大幅运动的场景中仍表现不佳。本文提出RelayGS，这是一种基于3DGS的新颖方法，专门用于表示和重建高度动态的场景。我们的RelayGS通过规范3D高斯和紧凑运动场学习完整的4D表示，包括三个阶段。首先，我们从所有帧中学习基础3DGS，忽略场景的时间变化，并使用可学习掩码将高度动态的前景与运动极小的背景分离。其次，我们复制第一阶段解耦出的前景高斯的多个副本，每个副本对应一个时间段，并使用每个时间段内由多帧构建的伪视图对其进行优化。这些高斯被称为接力高斯，充当显式接力节点，将大规模运动轨迹简化并分解为更小、可管理的片段。最后，我们联合学习场景的时间运动，并细化从前两个阶段学到的规范高斯。我们在两个具有大规模复杂运动的动态场景数据集上进行了充分实验，其中我们的RelayGS在PSNR上优于最先进方法超过1 dB，并以更加完整和连贯的方式成功重建了真实篮球比赛场景，而以往方法通常难以捕捉球员的复杂运动。代码将在https://github.com/gqk/RelayGS公开发布。

英文摘要：

Reconstructing dynamic scenes with large-scale and complex motions remains a significant challenge. Recent techniques like Neural Radiance Fields and 3D Gaussian Splatting (3DGS) have shown promise but still struggle with scenes involving substantial movement. This paper proposes RelayGS, a novel method based on 3DGS, specifically designed to represent and reconstruct highly dynamic scenes. Our RelayGS learns a complete 4D representation with canonical 3D Gaussians and a compact motion field, consisting of three stages. First, we learn a fundamental 3DGS from all frames, ignoring temporal scene variations, and use a learnable mask to separate the highly dynamic foreground from the minimally moving background. Second, we replicate multiple copies of the decoupled foreground Gaussians from the first stage, each corresponding to a temporal segment, and optimize them using pseudo-views constructed from multiple frames within each segment. These Gaussians, termed Relay Gaussians, act as explicit relay nodes, simplifying and breaking down large-scale motion trajectories into smaller, manageable segments. Finally, we jointly learn the scene's temporal motion and refine the canonical Gaussians learned from the first two stages. We conduct thorough experiments on two dynamic scene datasets featuring large and complex motions, where our RelayGS outperforms state-of-the-arts by more than 1 dB in PSNR, and successfully reconstructs real-world basketball game scenes in a much more complete and coherent manner, whereas previous methods usually struggle to capture the complex motion of players. Code will be publicly available at https://github.com/gqk/RelayGS

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。