我们提出了一套系统，能够从动态场景的随意拍摄单目视频中准确、快速且鲁棒地估计相机参数和深度图。大多数传统的运动恢复结构（Structure from Motion）和单目SLAM技术都假设输入视频以静态场景为主，且存在大量视差。在不满足这些条件时，这类方法往往会产生错误的估计结果。近期基于神经网络的方法试图克服这些挑战，但此类方法要么计算成本高昂，要么在相机运动不受控或视场未知的动态视频上运行时鲁棒性很差。我们展示了深度视觉SLAM框架出人意料的有效性：通过对其训练和推理方案进行精心调整，该系统能够适配具有无约束相机路径的复杂动态场景真实世界视频，包括相机视差很小的视频。在合成视频和真实视频上开展的大量实验表明，与现有及同期研究工作相比，我们的系统在相机位姿和深度估计方面准确性和鲁棒性显著提升，且运行速度更快或与现有方法相当。可在我们的项目页面查看交互式结果：https://mega-sam.github.io/
英文摘要：
We present a system that allows for accurate, fast, and robust estimation of camera parameters and depth maps from casual monocular videos of dynamic scenes. Most conventional structure from motion and monocular SLAM techniques assume input videos that feature predominantly static scenes with large amounts of parallax. Such methods tend to produce erroneous estimates in the absence of these conditions. Recent neural network-based approaches attempt to overcome these challenges; however, such methods are either computationally expensive or brittle when run on dynamic videos with uncontrolled camera motion or unknown field of view. We demonstrate the surprising effectiveness of a deep visual SLAM framework: with careful modifications to its training and inference schemes, this system can scale to real-world videos of complex dynamic scenes with unconstrained camera paths, including videos with little camera parallax. Extensive experiments on both synthetic and real videos demonstrate that our system is significantly more accurate and robust at camera pose and depth estimation when compared with prior and concurrent work, with faster or comparable running times. See interactive results on our project page: https://mega-sam.github.io/
DreaMo: Articulated 3D Reconstruction From A Single Casual Video
National Tsing Hua University（国立清华大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Merced（加州大学默塞德分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Amazon（亚马逊）
针对单段随手拍摄互联网视频视角不全的问题，本文提出 DreaMo，结合视角条件扩散先验、定制正则化及骨架生成策略，实现高质量铰接式 3D 形状与可解释骨架重建。
AASS research center, Örebro University, Sweden（奥雷布罗大学AASS研究中心）；Czech Technical University in Prague（布拉格捷克技术大学）；Technical University of Munich, Chair: Perception for Intelligent Systems（慕尼黑技术大学感知智能系统教研室）
International Institute of Information Technology (IIIT)（国际信息技术研究所）；Université Laval（拉瓦尔大学）
We present PanoHDR-NeRF, a neural representation of the full HDR radiance field of an indoor scene, and a pipeline to capture it casually, without elaborate setups or complex capture protocols. First, a user captures a low dynamic range (LDR) omnidirectional video of the scene by freely waving an off-the-shelf camera around the scene. Then, an LDR2HDR network uplifts the captured LDR frames to HDR, which are used to train a tailored NeRF++ model. The resulting PanoHDR-NeRF can render full HDR images from any location of the scene. Through experiments on a novel test dataset of real scenes with…
Computational Media Innovation Centre, Victoria University of Wellington（惠灵顿维多利亚大学计算媒体创新中心）；School of Engineering and Computer Science, Victoria University of Wellington（惠灵顿维多利亚大学工程与计算机科学学院）
Six degrees-of-freedom (6-DoF) video provides telepresence by enabling users to move around in the captured scene with a wide field of regard. Compared to methods requiring sophisticated camera setups, the image-based rendering method based on photogrammetry can work with images captured with any poses, which is more suitable for casual users. However, existing image-based rendering methods are based on perspective images. When used to reconstruct 6-DoF views, it often requires capturing hundreds of images, making data capture a tedious and time-consuming process. In contrast to traditional pe…
We present an algorithm for generating novel views at arbitrary viewpoints and any input time step given a monocular video of a dynamic scene. Our work builds upon recent advances in neural implicit representation and uses continuous and differentiable functions for modeling the time-varying structure and the appearance of the scene. We jointly train a time-invariant static NeRF and a time-varying dynamic NeRF, and learn how to blend the results in an unsupervised manner. However, learning this implicit function from a single video is highly ill-posed (with infinitely many solutions that match…
University of Washington（华盛顿大学）；Google Research（谷歌研究院）
We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to Ne…
University of California, Berkeley（加州大学伯克利分校）；Facebook Research（Facebook研究院）；Chapman University（查普曼大学）
In cinema, large camera lenses create beautiful shallow depth of field (DOF), but make focusing difficult and expensive. Accurate cinema focus usually relies on a script and a person to control focus in realtime. Casual videographers often crave cinematic focus, but fail to achieve it. We either sacrifice shallow DOF, as in smartphone videos; or we struggle to deliver accurate focus, as in videos from larger cameras. This paper is about a new approach in the pursuit of cinematic focus for casual videography. We present a system that synthetically renders refocusable video from a deep DOF video…
Disney Research Zurich（苏黎世迪士尼研究院）；TU Braunschweig（布伦瑞克工业大学）
Many compelling video processing effects can be achieved if per-pixel depth information and 3D camera calibrations are known. However, the success of such methods is highly dependent on the accuracy of this "scene-space" information. We present a novel, sampling-based framework for processing video that enables high-quality scene-space video effects in the presence of inevitable errors in depth and camera pose estimation. Instead of trying to improve the explicit 3D scene representation, the key idea of our method is to exploit the high redundancy of approximate scene information that arises d…
4DAnyone: Create Anyone in 4D from a Casual Monocular Video
State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）；Robbyant；Chinese University of Hong Kong（香港中文大学）；Ant Group（蚂蚁集团）；Hong Kong University of Science and Technology（香港科技大学）
We present EgoNeRF, a practical solution to reconstruct large-scale real-world environments for VR assets. Given a few seconds of casually captured 360 video, EgoNeRF can efficiently build neural radiance fields which enable high-quality rendering from novel viewpoints. Motivated by the recent acceleration of NeRF using feature grids, we adopt spherical coordinate instead of conventional Cartesian coordinate. Cartesian feature grid is inefficient to represent large-scale unbounded scenes because it has a spatially uniform resolution, regardless of distance from viewers. The spherical parameter…
Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；University of Electronic Science and Technology of China（电子科技大学）；Zhejiang University（浙江大学）；Kuaishou Technology（快手科技）；ByteDance（字节跳动）
Neural Radiance Fields (NeRFs) encode the radiance in a scene parameterized by the scene's plenoptic function. This is achieved by using an MLP together with a mapping to a higher-dimensional space, and has been proven to capture scenes with a great level of detail. Naturally, the same parameterization can be used to encode additional properties of the scene, beyond just its radiance. A particularly interesting property in this regard is the semantic decomposition of the scene. We introduce a novel technique for semantic soft decomposition of neural radiance fields (named SSDNeRF) which jointl…
