Recent advances in 4D generation mainly focus on generating 4D content by distilling pre-trained text or single-view image-conditioned models. It is inconvenient for them to take advantage of various off-the-shelf 3D assets with multi-view attributes, and their results suffer from spatiotemporal inconsistency owing to the inherent ambiguity in the supervision signals. In this work, we present Animate3D, a novel framework for animating any static 3D model. The core idea is two-fold: 1) We propose a novel multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, which is trained on our presented large-scale multi-view video dataset (MV-Video). 2) Based on MV-VDM, we introduce a framework combining reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the multi-view video diffusion priors for animating 3D objects. Specifically, for MV-VDM, we design a new spatiotemporal attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. Additionally, we leverage the static 3D model's multi-view renderings as conditions to preserve its identity. For animating 3D models, an effective two-stage pipeline is proposed: we first reconstruct motions directly from generated multi-view videos, followed by the introduced 4D-SDS to refine both appearance and motion. Benefiting from accurate motion learning, we could achieve straightforward mesh animation. Qualitative and quantitative experiments demonstrate that Animate3D significantly outperforms previous approaches. Data, code, and models will be open-released.
Animating the Past: Reconstruct Trilobite via Video Generation
AI Lab Yishi Inc.(杭州意科技有限公司AI实验室);The International Department Experimental High School Attached to Beijing Normal University(北京师范大学附属实验中学国际部);School of Ocean Sciences China University of Geoscience(中国地质大学海洋科学学院)
Animating Arbitrary Objects via Deep Motion Transfer
DISI, University of Trento(特伦托大学信息工程与计算机科学系(DISI));Snap Inc.(Snap公司);Fondazione Bruno Kessler (FBK)(布鲁诺·凯斯勒基金会(FBK));Huawei Technologies Ireland(华为技术爱尔兰公司);University of Trento(特伦托大学)
EchoFlow: A Foundation Model for Cardiac Ultrasound Image and Video Generation
Imperial College London(帝国理工学院);University of Birmingham(伯明翰大学);Ultromics Ltd.(奥特罗米克斯有限公司);University of Oxford(牛津大学);Friedrich-Alexander-Universität Erlangen-Nürnberg(埃尔朗根-纽伦堡弗里德里希-亚历山大大学)
SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models
PRIS, School of Artificial Intelligence, Beijing University of Posts(PRIS人工智能学院,北京邮电大学);SketchX, CVSSP, University of Surrey(SketchX,计算机视觉与模式识别研究所,萨里大学)
Drive Any Mesh: 4D Latent Diffusion for Mesh Deformation from Video
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University(虚拟现实技术与系统国家重点实验室,北京航空航天大学);School of Electronic and Computer Engineering, Peking University(电子与计算机工程学院,北京大学);Baidu VIS(百度VIS)
IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Kling Team, Kuaishou Technology(快手技术 Kling 团队);Cardiff University(卡迪夫大学)
Sora as a World Model? A Complete Survey on Text-to-Video Generation
University of Electronic Science and Technology of China(电子科技大学);Kyung Hee University(韩国庆熙大学);The Hong Kong Polytechnic University(香港理工大学);Harbin Institute of Technology Shenzhen(哈尔滨工业大学深圳学院);Nota Inc.(Nota公司);Tongji University(同济大学)
Human Video Generation from a Single Image with 3D Pose and View Control
Electrical Engineering and Computer Science, University of California, Merced, CA 95343, United States.(电子工程与计算机科学系,加州大学默塞德分校);Stability AI, Los Angeles, CA 90067, United States.(Stability AI)
MV-Performer: Taming Video Diffusion Model for Faithful and Synchronized Multi-view Performer Synthesis
Great Bay University(大亚湾大学);School of Artificial Intelligence, Shenzhen University(深圳大学人工智能学院);Guangdong Provincial Key Laboratory of Future Networks of Intelligence(广东省未来网络智能化重点实验室)