arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.14939cs.CV
收藏

STAG4D:时空锚定的生成式 4D 高斯

STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians

Nanjing University(南京大学);Institution of Automation, Chinese Academy of Science(中国科学院自动化研究所);Fudan University(复旦大学)

Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, Yao Yao

AI总结:

提出STAG4D框架,结合预训练扩散模型与动态3D高斯泼溅,通过多视角序列初始化、时间锚点融合策略和自适应致密化的分数蒸馏采样,在无需微调的情况下实现了高保真且时空一致的4D生成。

摘要:

预训练扩散模型和 3D 生成领域的最新进展激发了人们对 4D 内容创作的兴趣。然而,实现具有时空一致性的高保真 4D 生成仍然是一个挑战。在这项工作中,我们提出了 STAG4D,这是一种将预训练扩散模型与动态 3D 高斯泼溅相结合以实现高保真 4D 生成的新颖框架。受 3D 生成技术的启发,我们利用多视角扩散模型来初始化锚定于输入视频帧的多视角图像,其中视频可以是真实世界捕获的,也可以是由视频扩散模型生成的。为确保多视角序列初始化的时间一致性,我们引入了一种简单而有效的融合策略,在自注意力计算中利用第一帧作为时间锚点。借助几乎一致的多视角序列,我们随后应用分数蒸馏采样来优化 4D 高斯点云。4D 高斯泼溅专门为生成任务而精心设计,其中提出了一种自适应致密化策略,以缓解不稳定的梯度,从而实现稳健优化。值得注意的是,所提出的流程不需要对扩散网络进行任何预训练或微调,为 4D 生成任务提供了一种更易于使用且实用的解决方案。大量实验表明,我们的方法在渲染质量、时空一致性和生成鲁棒性方面优于先前的 4D 生成工作,为从文本、图像和视频等多样输入进行 4D 生成树立了新的最先进水平。

英文摘要:

Recent progress in pre-trained diffusion models and 3D generation have spurred interest in 4D content creation. However, achieving high-fidelity 4D generation with spatial-temporal consistency remains a challenge. In this work, we propose STAG4D, a novel framework that combines pre-trained diffusion models with dynamic 3D Gaussian splatting for high-fidelity 4D generation. Drawing inspiration from 3D generation techniques, we utilize a multi-view diffusion model to initialize multi-view images anchoring on the input video frames, where the video can be either real-world captured or generated by a video diffusion model. To ensure the temporal consistency of the multi-view sequence initialization, we introduce a simple yet effective fusion strategy to leverage the first frame as a temporal anchor in the self-attention computation. With the almost consistent multi-view sequences, we then apply the score distillation sampling to optimize the 4D Gaussian point cloud. The 4D Gaussian spatting is specially crafted for the generation task, where an adaptive densification strategy is proposed to mitigate the unstable Gaussian gradient for robust optimization. Notably, the proposed pipeline does not require any pre-training or fine-tuning of diffusion networks, offering a more accessible and practical solution for the 4D generation task. Extensive experiments demonstrate that our method outperforms prior 4D generation works in rendering quality, spatial-temporal consistency, and generation robustness, setting a new state-of-the-art for 4D generation from diverse inputs, including text, image, and video.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑