arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.11398cs.CV
收藏

Animate3D:使用多视图视频扩散模型为任意3D模型制作动画

Animate3D: Animating Any 3D Model with Multi-view Video Diffusion

DAMO Academy, Alibaba Group(阿里巴巴达摩院);Hupan Lab(湖畔实验室)

Yanqin Jiang, Chaohui Yu, Chenjie Cao, Fan Wang, Weiming Hu, Jin Gao

AI总结:

本文提出Animate3D框架,通过多视图视频扩散模型MV-VDM及重建与4D-SDS两阶段流程,为任意静态3D模型生成时空一致的动画。

摘要:

近期4D生成的进展主要集中于通过蒸馏预训练文本或单视图图像条件模型来生成4D内容。这些方法不便利用各类现成的、具有多视图属性的3D资产,并且由于监督信号中固有的模糊性,其结果会出现时空不一致问题。在本文中,我们提出Animate3D,这是一个用于为任意静态3D模型制作动画的新颖框架。核心思想包含两点:1)我们提出一种新颖的多视图视频扩散模型(MV-VDM),该模型以静态3D对象的多视图渲染图为条件,并在我们所提出的大规模多视图视频数据集(MV-Video)上进行训练。2)基于MV-VDM,我们引入一个结合重建与4D Score Distillation Sampling(4D-SDS,4D分数蒸馏采样)的框架,以利用多视图视频扩散先验为3D对象制作动画。具体而言,针对MV-VDM,我们设计了一个新的时空注意力模块,通过整合3D扩散模型和视频扩散模型来增强空间与时间一致性。此外,我们利用静态3D模型的多视图渲染图作为条件以保留其身份特征。为了给3D模型制作动画,我们提出了有效的两阶段流程:首先直接从生成的多视图视频中重建运动,随后采用所引入的4D-SDS同时细化外观和运动。得益于准确的运动学习,我们能够实现直接的网格动画。定性和定量实验表明,Animate3D显著优于以往方法。数据、代码和模型将公开发布。

英文摘要:

Recent advances in 4D generation mainly focus on generating 4D content by distilling pre-trained text or single-view image-conditioned models. It is inconvenient for them to take advantage of various off-the-shelf 3D assets with multi-view attributes, and their results suffer from spatiotemporal inconsistency owing to the inherent ambiguity in the supervision signals. In this work, we present Animate3D, a novel framework for animating any static 3D model. The core idea is two-fold: 1) We propose a novel multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, which is trained on our presented large-scale multi-view video dataset (MV-Video). 2) Based on MV-VDM, we introduce a framework combining reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the multi-view video diffusion priors for animating 3D objects. Specifically, for MV-VDM, we design a new spatiotemporal attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. Additionally, we leverage the static 3D model's multi-view renderings as conditions to preserve its identity. For animating 3D models, an effective two-stage pipeline is proposed: we first reconstruct motions directly from generated multi-view videos, followed by the introduced 4D-SDS to refine both appearance and motion. Benefiting from accurate motion learning, we could achieve straightforward mesh animation. Qualitative and quantitative experiments demonstrate that Animate3D significantly outperforms previous approaches. Data, code, and models will be open-released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑