arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2103.02597cs.CVcs.GR
收藏

基于神经网络的多视点三维视频合成

Neural 3D Video Synthesis from Multi-view Video

University of Southern California(南加州大学);Reality Labs Research(现实实验室研究部);Meta

Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, Zhaoyang Lv

摘要:

我们提出了一种新的3D视频合成方法,能够表示多视图视频记录的动态现实世界的场景在一个紧凑的,但表现力的表示,使高质量的视图合成和运动插值。我们的方法将静态神经辐射场的高质量和紧凑性带到了一个新的方向:无模型的动态设置。在我们的方法的核心是一个新的时间条件下的神经辐射场,表示场景动态使用一组紧凑的潜在代码。我们能够显着提高训练速度和感知质量的生成图像的一种新的分层训练方案结合射线重要性采样。我们学习的表示是非常紧凑的,能够表示18个摄像头的10秒30 FPS多视图视频记录,模型大小仅为28MB。我们证明了我们的方法可以在超过1K的分辨率下渲染高保真广角新颖的视图,即使是复杂和动态的场景。我们进行了广泛的定性和定量评估,表明我们的方法优于最先进的方法。项目网站:www.example.com。

英文摘要:

We propose a novel approach for 3D video synthesis that is able to represent multi-view video recordings of a dynamic real-world scene in a compact, yet expressive representation that enables high-quality view synthesis and motion interpolation. Our approach takes the high quality and compactness of static neural radiance fields in a new direction: to a model-free, dynamic setting. At the core of our approach is a novel time-conditioned neural radiance field that represents scene dynamics using a set of compact latent codes. We are able to significantly boost the training speed and perceptual quality of the generated imagery by a novel hierarchical training scheme in combination with ray importance sampling. Our learned representation is highly compact and able to represent a 10 second 30 FPS multiview video recording by 18 cameras with a model size of only 28MB. We demonstrate that our method can render high-fidelity wide-angle novel views at over 1K resolution, even for complex and dynamic scenes. We perform an extensive qualitative and quantitative evaluation that shows that our approach outperforms the state of the art. Project website: https://neural-3d-video.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑