arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.16611cs.CVcs.AIcs.LG
收藏

一种从单张图像生成3D世界的方案

A Recipe for Generating 3D Worlds From a Single Image

Meta Reality Labs(元宇宙实验室)

Katja Schwarz, Denys Rozumnyi, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder

AI总结:

本研究提出一种从单张图像生成3D世界的方案,将任务转化为2D修复模型的上下文学习问题,经全景生成、深度提升及点云引导的区域填补步骤,仅需少量训练,在多项指标上优于现有基于视频合成的方法,可生成适配VR的高质量3D环境。

摘要:

我们提出一种从单张图像生成沉浸式3D世界的方案,将该任务构建为2D修复模型的上下文学习问题。该方法所需训练极少,且利用现有生成模型。我们的流程包含两个步骤:使用预训练扩散模型生成连贯的全景图,再通过度量深度估计器将其提升为3D结构。随后,我们以渲染的点云为条件约束修复模型,填补未观测区域,仅需少量微调。在合成图像与真实图像上的测试表明,我们的方法能生成适用于VR显示的高质量3D环境。通过从一开始就显式建模生成环境的3D结构,我们的方法在多项定量图像质量指标上始终优于基于视频合成的现有最优方法。项目页面:https://katjaschwarz.github.io/worlds/

英文摘要:

We introduce a recipe for generating immersive 3D worlds from a single image by framing the task as an in-context learning problem for 2D inpainting models. This approach requires minimal training and uses existing generative models. Our process involves two steps: generating coherent panoramas using a pre-trained diffusion model and lifting these into 3D with a metric depth estimator. We then fill unobserved regions by conditioning the inpainting model on rendered point clouds, requiring minimal fine-tuning. Tested on both synthetic and real images, our method produces high-quality 3D environments suitable for VR display. By explicitly modeling the 3D structure of the generated environment from the start, our approach consistently outperforms state-of-the-art, video synthesis-based methods along multiple quantitative image quality metrics. Project Page: https://katjaschwarz.github.io/worlds/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑