arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.07452cs.CVcs.MM
收藏

Hi3D: 基于视频扩散模型的高分辨率图像到3D生成研究

Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models

HiDream.ai Inc.(嗨梦人工智能公司);Singapore Management University(新加坡管理大学)

Haibo Yang, Yang Chen, Yingwei Pan, Ting Yao, Zhineng Chen, Chong-Wah Ngo, Tao Mei

AI总结:

针对现有图像到3D生成方法难以产出高分辨率、多视角一致纹理图像的问题,提出基于视频扩散的Hi3D范式,通过赋予模型3D感知先验、细化纹理及3D重建,实现了效果更优的高分辨率图像到3D生成。

摘要:

尽管图像到3D生成已取得巨大进展,现有方法仍难以生成具备高分辨率纹理细节的多视角一致图像,尤其在缺乏3D感知的2D扩散范式中问题更为突出。本研究提出高分辨率图像到3D模型Hi3D,这是一种全新的基于视频扩散的范式,将单图像到多视角图像的生成重新定义为3D感知的序列图像生成(即轨道视频生成)。该方法挖掘了视频扩散模型中潜在的时序一致性知识,这类知识能很好地泛化到3D生成中多视角间的几何一致性上。技术层面,Hi3D首先为预训练视频扩散模型赋予3D感知先验(相机位姿条件),生成具有低分辨率纹理细节的多视角图像;随后学习一个3D感知的视频到视频细化器,进一步将多视角图像提升至高分辨率纹理细节。这些高分辨率多视角图像还会通过3D Gaussian Splatting(3D高斯溅射)补充新视角,最终借助3D重建得到高保真网格。在新视角合成和单视角重建任务上的大量实验表明,Hi3D能够生成多视角一致性更优、纹理细节高度丰富的图像。源代码和数据可在https://github.com/yanghb22-fdu/Hi3D-Official获取。

英文摘要:

Despite having tremendous progress in image-to-3D generation, existing methods still struggle to produce multi-view consistent images with high-resolution textures in detail, especially in the paradigm of 2D diffusion that lacks 3D awareness. In this work, we present High-resolution Image-to-3D model (Hi3D), a new video diffusion based paradigm that redefines a single image to multi-view images as 3D-aware sequential image generation (i.e., orbital video generation). This methodology delves into the underlying temporal consistency knowledge in video diffusion model that generalizes well to geometry consistency across multiple views in 3D generation. Technically, Hi3D first empowers the pre-trained video diffusion model with 3D-aware prior (camera pose condition), yielding multi-view images with low-resolution texture details. A 3D-aware video-to-video refiner is learnt to further scale up the multi-view images with high-resolution texture details. Such high-resolution multi-view images are further augmented with novel views through 3D Gaussian Splatting, which are finally leveraged to obtain high-fidelity meshes via 3D reconstruction. Extensive experiments on both novel view synthesis and single view reconstruction demonstrate that our Hi3D manages to produce superior multi-view consistency images with highly-detailed textures. Source code and data are available at \url{https://github.com/yanghb22-fdu/Hi3D-Official}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑