arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.02148cs.CV
收藏

Diffusion²:通过视频与多视图扩散模型的分数组合实现动态3D内容生成

Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models

Fudan University(复旦大学)

Zeyu Yang, Zijie Pan, Chun Gu, Li Zhang

AI总结:

本文提出 Diffusion² 框架,通过组合视频与多视图扩散模型的分数并采用插值步方差缩减采样,直接生成密集多视图多帧图像以重建4D内容。

摘要:

近期3D生成的进展主要由具备3D感知能力的图像扩散模型的提升所推动。这些模型在互联网规模的图像数据上进行预训练,并在大规模3D数据上进行微调,从而能够生成高度一致的多视图图像。然而,由于缺乏同步的多视图视频数据,将这一范式直接适配到4D生成仍然具有挑战性。尽管如此,现有的视频和3D数据足以分别训练视频扩散模型与多视图扩散模型,使它们分别提供令人满意的动态先验和几何先验。为同时利用二者优势,本文提出 Diffusion²,这是一个用于动态3D内容创作的新颖框架,能够协调来自这些模型的几何一致性知识与时间平滑性知识,以直接采样密集的多视图多帧图像,并利用这些图像优化连续4D表示。具体而言,我们基于目标图像阵列的概率结构,通过对预训练视频扩散模型和多视图扩散模型进行分数组合,设计了一种简单而有效的去噪策略。为缓解两个异构分数之间潜在的冲突,我们进一步引入通过插值步骤实现的方差缩减采样,以促进平滑、稳定的生成。得益于所提出图像生成流程的高度并行性以及现代4D重建管线的效率,我们的框架可在数分钟内生成4D内容。值得注意的是,我们的方法规避了对昂贵且难以扩展的4D数据的依赖,因此有潜力受益于基础视频扩散模型和多视图扩散模型的规模化扩展。大量实验证明,所提出框架在各种类型条件下生成高度无缝且一致的4D资产方面具有有效性。

英文摘要:

Recent advancements in 3D generation are predominantly propelled by improvements in 3D-aware image diffusion models. These models are pretrained on Internet-scale image data and fine-tuned on massive 3D data, offering the capability of producing highly consistent multi-view images. However, due to the scarcity of synchronized multi-view video data, it remains challenging to adapt this paradigm to 4D generation directly. Despite that, the available video and 3D data are adequate for training video and multi-view diffusion models separately that can provide satisfactory dynamic and geometric priors respectively. To take advantage of both, this paper presents Diffusion$^2$, a novel framework for dynamic 3D content creation that reconciles the knowledge about geometric consistency and temporal smoothness from these models to directly sample dense multi-view multi-frame images which can be employed to optimize continuous 4D representation. Specifically, we design a simple yet effective denoising strategy via score composition of pretrained video and multi-view diffusion models based on the probability structure of the target image array. To alleviate the potential conflicts between two heterogeneous scores, we further introduce variance-reducing sampling via interpolated steps, facilitating smooth and stable generation. Owing to the high parallelism of the proposed image generation process and the efficiency of the modern 4D reconstruction pipeline, our framework can generate 4D content within few minutes. Notably, our method circumvents the reliance on expensive and hard-to-scale 4D data, thereby having the potential to benefit from the scaling of the foundation video and multi-view diffusion models. Extensive experiments demonstrate the efficacy of our proposed framework in generating highly seamless and consistent 4D assets under various types of conditions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑