arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.07770cs.CVcs.LG
收藏

从图像到场景:从百万个360°视频中学习想象世界

From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos

University of Washington(华盛顿大学);Toyota Technological Institute at Chicago(芝加哥丰田技术学院);Harvard University(哈佛大学);Allen Institute for AI(艾伦人工智能研究所);Cornell University(康奈尔大学)

Matthew Wallingford, Anand Bhattad, Aditya Kusupati, Vivek Ramanujan, Matt Deitke, Sham Kakade, Aniruddha Kembhavi, Roozbeh Mottaghi, Wei-Chiu Ma, Ali Farhadi

AI总结:

针对真实世界3D场景理解缺乏大规模数据的问题,研究团队构建百万级360°视频数据集360-1M,训练扩散模型Odin,实现可移动相机的真实场景新视图生成,提升相关基准性能。

摘要:

对物体与场景的三维(3D)理解是人类与世界交互能力的核心,也一直是计算机视觉、图形学和机器人领域的活跃研究方向。大规模合成数据集与以物体为中心的3D数据集已被证明可有效训练具备物体3D理解能力的模型,但由于缺乏大规模数据,将类似方法应用于真实世界物体与场景存在难度。视频是真实世界3D数据的潜在来源,但大规模获取同一内容的多样且对应视图已被证明十分困难。此外,标准视频的视点在拍摄时就已固定,限制了从更多样、更具实用价值的视角访问场景的能力。我们认为,大规模360°视频可解决这些局限,提供可扩展的多视图对应帧。本文提出360-1M这一360°视频数据集,以及一种可大规模高效查找多样视点对应帧的流程。我们在360-1M上训练了基于扩散的模型Odin。借助迄今最大规模的真实世界多视图数据集,Odin能够自由生成真实世界场景的新视图。与以往方法不同,Odin可让相机在环境中移动,使模型能够推断场景的几何结构与布局。此外,我们还证明该模型在标准新视图合成与3D重建基准测试上的性能有所提升。

英文摘要:

Three-dimensional (3D) understanding of objects and scenes play a key role in humans' ability to interact with the world and has been an active area of research in computer vision, graphics, and robotics. Large scale synthetic and object-centric 3D datasets have shown to be effective in training models that have 3D understanding of objects. However, applying a similar approach to real-world objects and scenes is difficult due to a lack of large-scale data. Videos are a potential source for real-world 3D data, but finding diverse yet corresponding views of the same content has shown to be difficult at scale. Furthermore, standard videos come with fixed viewpoints, determined at the time of capture. This restricts the ability to access scenes from a variety of more diverse and potentially useful perspectives. We argue that large scale 360 videos can address these limitations to provide: scalable corresponding frames from diverse views. In this paper, we introduce 360-1M, a 360 video dataset, and a process for efficiently finding corresponding frames from diverse viewpoints at scale. We train our diffusion-based model, Odin, on 360-1M. Empowered by the largest real-world, multi-view dataset to date, Odin is able to freely generate novel views of real-world scenes. Unlike previous methods, Odin can move the camera through the environment, enabling the model to infer the geometry and layout of the scene. Additionally, we show improved performance on standard novel view synthesis and 3D reconstruction benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑