In this work, we address the problem of refining the geometry of local image features from multiple views without known scene or camera geometry. Current approaches to local feature detection are inherently limited in their keypoint localization accuracy because they only operate on a single view. This limitation has a negative impact on downstream tasks such as Structure-from-Motion, where inaccurate keypoints lead to large errors in triangulation and camera localization. Our proposed method naturally complements the traditional feature extraction and matching paradigm. We first estimate local geometric transformations between tentative matches and then optimize the keypoint locations over multiple views jointly according to a non-linear least squares formulation. Throughout a variety of experiments, we show that our method consistently improves the triangulation and camera localization performance for both hand-crafted and learned local features.
G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images
South China University of Technology（华南理工大学）；The University of Adelaide（阿德莱德大学）；Guangzhou Shiyuan Electronics Co., Ltd（广州视源电子科技股份有限公司）；Pazhou Lab（琶洲实验室）
Recent multi-view multimedia applications struggle between high-resolution (HR) visual experience and storage or bandwidth constraints. Therefore, this paper proposes a Multi-View Image Super-Resolution (MVISR) task. It aims to increase the resolution of multi-view images captured from the same scene. One solution is to apply image or video super-resolution (SR) methods to reconstruct HR results from the low-resolution (LR) input view. However, these methods cannot handle large-angle transformations between views and leverage information in all multi-view images. To address these problems, we …
School of Communication Engineering, Hangzhou Dianzi University（杭州电子科技大学通信工程学院）
We present to recover the complete 3D facial geometry from a single depth view by proposing an Attention Guided Generative Adversarial Networks (AGGAN). In contrast to existing work which normally requires two or more depth views to recover a full 3D facial geometry, the proposed AGGAN is able to generate a dense 3D voxel grid of the face from a single unconstrained depth view. Specifically, AGGAN encodes the 3D facial geometry within a voxel space and utilizes an attention-guided GAN to model the illposed 2.5D depth-3D mapping. Multiple loss functions, which enforce the 3D facial geometry con…
Robert Bosch GmbH（罗伯特·博世有限公司）；Goethe University Frankfurt（法兰克福歌德大学）；NTNU Trondheim（特隆赫姆挪威科技大学）
Existing 3D scene flow estimation methods provide the 3D geometry and 3D motion of a scene and gain a lot of interest, for example in the context of autonomous driving. These methods are traditionally based on a temporal series of stereo images. In this paper, we propose a novel monocular 3D scene flow estimation method, called Mono-SF. Mono-SF jointly estimates the 3D structure and motion of the scene by combining multi-view geometry and single-view depth information. Mono-SF considers that the scene flow should be consistent in terms of warping the reference image in the consecutive image ba…
MetaView: Monocular Novel View Synthesis with Scale-Aware Implicit Geometry Priors
Nanyang Technological University（南洋理工大学）；Kolors Team, Kuaishou Technology（快手科技色彩团队）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Multi-view depth estimation methods typically require the computation of a multi-view cost-volume, which leads to huge memory consumption and slow inference. Furthermore, multi-view matching can fail for texture-less surfaces, reflective surfaces and moving objects. For such failure modes, single-view depth estimation methods are often more reliable. To this end, we propose MaGNet, a novel framework for fusing single-view depth probability with multi-view geometry, to improve the accuracy, robustness and efficiency of multi-view depth estimation. For each frame, MaGNet estimates a single-view …
Center for Research in Computer Vision (CRCV), University of Central Florida（中佛罗里达大学计算机视觉研究中心（CRCV））
We address the problem of generating images across two drastically different views, namely ground (street) and aerial (overhead) views. Image synthesis by itself is a very challenging computer vision task and is even more so when generation is conditioned on an image in another view. Due the difference in viewpoints, there is small overlapping field of view and little common content between these two views. Here, we try to preserve the pixel information between the views so that the generated image is a realistic representation of cross view input image. For this, we propose to use homography …
More Than Where You Are: Learning Semantics, Structure, and Geometry from Cross-View Localization
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Aerospace Control Center（北京航天飞行控制中心）
This paper tackles the problem of novel view synthesis from a single image. In particular, we target real-world scenes with rich geometric structure, a challenging task due to the large appearance variations of such scenes and the lack of simple 3D models to represent them. Modern, learning-based approaches mostly focus on appearance to synthesize novel views and thus tend to generate predictions that are inconsistent with the underlying scene structure. By contrast, in this paper, we propose to exploit the 3D geometry of the scene to synthesize a novel view. Specifically, we approximate a rea…
We present a self-supervised learning algorithm for 3D human pose estimation of a single person based on a multiple-view camera system and 2D body pose estimates for each view. To train our model, represented by a deep neural network, we propose a four-loss function learning algorithm, which does not require any 2D or 3D body pose ground-truth. The proposed loss functions make use of the multiple-view geometry to reconstruct 3D body pose estimates and impose body pose constraints across the camera views. Our approach utilizes all available camera views during training, while the inference is s…
Hong Kong University of Science and Technology（香港科技大学）；Everest Innovation Technology（珠峰创新科技）
Recent learning-based approaches, in which models are trained by single-view images have shown promising results for monocular 3D face reconstruction, but they suffer from the ill-posed face pose and depth ambiguity issue. In contrast to previous works that only enforce 2D feature constraints, we propose a self-supervised training architecture by leveraging the multi-view geometry consistency, which provides reliable constraints on face pose and depth estimation. We first propose an occlusion-aware view synthesis method to apply multi-view geometry consistency to self-supervised learning. Then…
We tackle the task of synthesizing novel views of an object given a few input images and associated camera viewpoints. Our work is inspired by recent 'geometry-free' approaches where multi-view images are encoded as a (global) set-latent representation, which is then used to predict the color for arbitrary query rays. While this representation yields (coarsely) accurate images corresponding to novel viewpoints, the lack of geometric reasoning limits the quality of these outputs. To overcome this limitation, we propose 'Geometry-biased Transformers' (GBTs) that incorporate geometric inductive b…
Università degli Studi di Parma（帕尔马大学）；VisLab Srl（VisLab有限责任公司）；Ambarella Inc（安霸公司）
We present a framework, called MVG-NeRF, that combines classical Multi-View Geometry algorithms and Neural Radiance Fields (NeRF) for image-based 3D reconstruction. NeRF has revolutionized the field of implicit 3D representations, mainly due to a differentiable volumetric rendering formulation that enables high-quality and geometry-aware novel view synthesis. However, the underlying geometry of the scene is not explicitly constrained during training, thus leading to noisy and incorrect results when extracting a mesh with marching cubes. To this end, we propose to leverage pixelwise depths and …
VGD: Visual Geometry Gaussian Splatting for Feed-Forward Surround-view Driving Reconstruction
Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology（广东省超高清沉浸媒体技术重点实验室）；School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Peng Cheng Laboratory（鹏城实验室）
MAGE: View-guided Point Cloud Completion with Efficient Modality Alignment and Adaptive Geometry Enhancement
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS研究室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；GIPSA-lab, Univ. Grenoble Alpes, CNRS, Grenoble INP（格勒诺布尔大学GIPSA实验室，法国国家科学研究中心，格勒诺布尔国立综合理工学院）
GeoSurDepth: Harnessing Foundation Model for Spatial Geometry Consistency-Oriented Self-Supervised Surround-View Depth Estimation
State Key Laboratory of Intelligent Green Vehicle and Mobility, School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China（1 智能绿色车辆与移动国家重点实验室，车辆与移动学院，清华大学，北京100084，中国）；California PATH, University of California, Berkeley, CA, United States（2 加州PATH，加州大学伯克利分校，加州，美国）
Australian National University（澳大利亚国立大学）；University of Oxford（牛津大学）；University of Technology Sydney（悉尼科技大学）
This paper presents a new approach for synthesizing a novel street-view panorama given an overhead satellite image. Taking a small satellite image patch as input, our method generates a Google's omnidirectional street-view type panorama, as if it is captured from the same geographical location as the center of the satellite patch. Existing works tackle this task as an image generation problem which adopts generative adversarial networks to implicitly learn the cross-view transformations, while ignoring the domain relevance. In this paper, we propose to explicitly establish the geometric corres…
Geo-EVS: Geometry-Conditioned Extrapolative View Synthesis for Autonomous Driving
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与运载学院）；State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（清华大学智能绿色车辆与移动国家重点实验室）；School of Science, Minzu University（中央民族大学理学院）；Chongqing Changan Automobile Co., Ltd.（重庆长安汽车股份有限公司）
Learning Fine-Grained Geometry for Sparse-View Splatting via Cascade Depth Loss
School of Computer Science The University of Sydney Sydney, Australia；School of Computer Science University of Bristol Bristol, United Kingdom；School of Science Edith Cowan University Perth, Australia
Nanyang Technological University（南洋理工大学）；A*STAR（新加坡科技研究局）；Huazhong University of Science and Technology（华中科技大学）
Recently, neural implicit surfaces learning by volume rendering has become popular for multi-view reconstruction. However, one key challenge remains: existing approaches lack explicit multi-view geometry constraints, hence usually fail to generate geometry consistent surface reconstruction. To address this challenge, we propose geometry-consistent neural implicit surfaces learning for multi-view reconstruction. We theoretically analyze that there exists a gap between the volume rendering integral and point-based signed distance function (SDF) modeling. To bridge this gap, we directly locate th…
Recovering detailed facial geometry from a set of calibrated multi-view images is valuable for its wide range of applications. Traditional multi-view stereo (MVS) methods adopt an optimization-based scheme to regularize the matching cost. Recently, learning-based methods integrate all these into an end-to-end neural network and show superiority of efficiency. In this paper, we propose a novel architecture to recover extremely detailed 3D faces within dozens of seconds. Unlike previous learning-based methods that regularize the cost volume via 3D CNN, we propose to learn an implicit function fo…
