arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.12010cs.CVcs.AIcs.GR
收藏

VideoMV:基于大型视频生成模型的一致多视角生成

VideoMV: Consistent Multi-View Generation Based on Large Video Generative Model

Institute for Intelligent Computing, Alibaba Group(阿里巴巴集团智能计算研究所);SSE, CUHKSZ(香港中文大学(深圳)数据科学学院);Peking University(北京大学);Fudan University(复旦大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Qi Zuo, Xiaodong Gu, Lingteng Qiu, Yuan Dong, Zhengyi Zhao, Weihao Yuan, Rui Peng, Siyu Zhu, Zilong Dong, Liefeng Bo, Qixing Huang

AI总结:

本文提出VideoMV框架,利用视频生成模型微调生成多视角图像,并通过3D感知去噪采样增强一致性,实现快速训练和高质量3D资产创建。

摘要:

基于文本或单图像提示生成多视角图像是创建3D内容的关键能力。关于此主题的两个基本问题是:我们使用什么数据进行训练,以及如何确保多视角一致性。本文介绍了一个新颖的框架,对这两个问题都做出了根本性贡献。与利用2D扩散模型中的图像进行训练不同,我们提出了一种密集一致的多视角生成模型,该模型从现成的视频生成模型中进行微调。来自视频生成模型的图像更适合多视角生成,因为生成它们的底层网络架构采用时间模块来强制帧一致性。此外,用于训练这些模型的视频数据集丰富多样,从而减小了训练与微调之间的域差距。为了增强多视角一致性,我们引入了3D感知去噪采样,该方法首先采用前馈重建模块来获取显式的全局3D模型,然后采用一种采样策略,有效地将全局3D模型渲染的图像纳入去噪采样循环中,以提高最终图像的多视角一致性。作为副产品,该模块还提供了一种在几秒钟内创建由3D高斯表示的3D资产的快速方法。我们的方法可以生成24个密集视图,并且在训练中比最先进的方法(4个GPU小时对比数千个GPU小时)收敛得快得多,同时具有可比的视觉质量和一致性。通过进一步微调,我们的方法在定量指标和视觉效果上都优于现有的最先进方法。我们的项目页面是aigc3d.github.io/VideoMV。

英文摘要:

Generating multi-view images based on text or single-image prompts is a critical capability for the creation of 3D content. Two fundamental questions on this topic are what data we use for training and how to ensure multi-view consistency. This paper introduces a novel framework that makes fundamental contributions to both questions. Unlike leveraging images from 2D diffusion models for training, we propose a dense consistent multi-view generation model that is fine-tuned from off-the-shelf video generative models. Images from video generative models are more suitable for multi-view generation because the underlying network architecture that generates them employs a temporal module to enforce frame consistency. Moreover, the video data sets used to train these models are abundant and diverse, leading to a reduced train-finetuning domain gap. To enhance multi-view consistency, we introduce a 3D-Aware Denoising Sampling, which first employs a feed-forward reconstruction module to get an explicit global 3D model, and then adopts a sampling strategy that effectively involves images rendered from the global 3D model into the denoising sampling loop to improve the multi-view consistency of the final images. As a by-product, this module also provides a fast way to create 3D assets represented by 3D Gaussians within a few seconds. Our approach can generate 24 dense views and converges much faster in training than state-of-the-art approaches (4 GPU hours versus many thousand GPU hours) with comparable visual quality and consistency. By further fine-tuning, our approach outperforms existing state-of-the-art methods in both quantitative metrics and visual effects. Our project page is aigc3d.github.io/VideoMV.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑