正交时空分布式转移用于4D生成
Orthogonal Spatial-temporal Distributional Transfer for 4D Generation
AI总结:
本文提出正交时空分布式转移机制,通过解耦空间和时间潜在变量提升4D生成的质量和一致性。
摘要:
在AIGC时代,生成高质量4D内容引起了越来越多的研究关注。不幸的是,当前4D合成研究严重受限于缺乏大规模4D数据集,阻碍了模型充分学习生成高质量4D内容所需的时空特征,从而阻碍了该领域的发展。为解决这一问题,我们提出了一种新的框架,将现有3D扩散模型中的丰富空间先验和视频扩散模型中的时间先验转移到增强4D合成。我们开发了空间-时间解耦的4D(STD-4D)扩散模型,通过解耦的空间和时间潜在变量生成4D-aware视频。为了促进最佳特征转移,我们设计了一种新的正交时空分布式转移(Orster)机制,其中时空特征分布被精心建模并注入到STD-4D扩散模型中。此外,在4D构建过程中,我们设计了空间-时间感知的HexPlane(ST-HexPlane)以整合转移的时空特征,从而提高4D变形和4D高斯特征建模。实验表明,我们的方法显著优于现有方法,实现了更优的时空一致性和更高质量的4D合成。
英文摘要:
In the AIGC era, generating high-quality 4D content has garnered increasing research attention. Unfortunately, current 4D synthesis research is severely constrained by the lack of large-scale 4D datasets, preventing models from adequately learning the critical spatial-temporal features necessary for high-quality 4D generation, thus hindering progress in this domain. To combat this, we propose a novel framework that transfers rich spatial priors from existing 3D diffusion models and temporal priors from video diffusion models to enhance 4D synthesis. We develop a spatial-temporal-disentangled 4D (STD-4D) Diffusion model, which synthesizes 4D-aware videos through disentangled spatial and temporal latents. To facilitate the best feature transfer, we design a novel Orthogonal Spatial-temporal Distributional Transfer (Orster) mechanism, where the spatiotemporal feature distributions are carefully modeled and injected into the STD-4D Diffusion. Furthermore, during the 4D construction, we devise a spatial-temporal-aware HexPlane (ST-HexPlane) to integrate the transferred spatiotemporal features, thereby improving 4D deformation and 4D Gaussian feature modeling. Experiments demonstrate that our method significantly outperforms existing approaches, achieving superior spatial-temporal consistency and higher-quality 4D synthesis.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。