We present a novel end-to-end framework named as GSNet (Geometric and Scene-aware Network), which jointly estimates 6DoF poses and reconstructs detailed 3D car shapes from single urban street view. GSNet utilizes a unique four-way feature extraction and fusion scheme and directly regresses 6DoF poses and shapes in a single forward pass. Extensive experiments show that our diverse feature extraction and fusion scheme can greatly improve model performance. Based on a divide-and-conquer 3D shape representation strategy, GSNet reconstructs 3D vehicle shape with great detail (1352 vertices and 2700 faces). This dense mesh representation further leads us to consider geometrical consistency and scene context, and inspires a new multi-objective loss function to regularize network training, which in turn improves the accuracy of 6D pose estimation and validates the merit of jointly performing both tasks. We evaluate GSNet on the largest multi-task ApolloCar3D benchmark and achieve state-of-the-art performance both quantitatively and qualitatively. Project page is available at https://lkeab.github.io/gsnet/.
University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）；Google Research（谷歌研究院）
Motivated by the astonishing capabilities of natural intelligent agents and inspired by theories from psychology, this paper explores the idea that perception gets coupled to 3D properties of the world via interaction with the environment. Existing works for depth estimation require either massive amounts of annotated training data or some form of hard-coded geometrical constraint. This paper explores a new approach to learning depth perception requiring neither of those. Specifically, we train a specialized global-local network architecture with what would be available to a robot interacting …
University of Oxford（牛津大学）；NAVER LABS Europe（欧洲NAVER实验室）
Self-supervision can dramatically cut back the amount of manually-labelled data required to train deep neural networks. While self-supervision has usually been considered for tasks such as image classification, in this paper we aim at extending it to geometry-oriented tasks such as semantic matching and part detection. We do so by building on several recent ideas in unsupervised landmark detection. Our approach learns dense distinctive visual descriptors from an unlabelled dataset of images using synthetic image transformations. It does so by means of a robust probabilistic formulation that ca…
GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervision
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；School of Electronics, Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；School of Transportation and Traffic Engineering, Central South University（中南大学交通运输工程学院）
UrbanGS: A Scalable and Efficient Architecture for Geometrically Accurate Large-Scene Reconstruction
Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；School of Software, Beihang University（北京航空航天大学软件学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）；Department of Computer Science, National University of Singapore（新加坡国立大学计算机科学系）；Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；Artificial Intelligence Research Center, Lobachevsky State University（洛夫奇夫斯基国立大学人工智能研究中心）
International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术学院）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）
3D LiDARs and 2D cameras are increasingly being used alongside each other in sensor rigs for perception tasks. Before these sensors can be used to gather meaningful data, however, their extrinsics (and intrinsics) need to be accurately calibrated, as the performance of the sensor rig is extremely sensitive to these calibration parameters. A vast majority of existing calibration techniques require significant amounts of data and/or calibration targets and human effort, severely impacting their applicability in large-scale production systems. We address this gap with CalibNet: a self-supervised …
Technical University of Munich（慕尼黑工业大学）；Tsinghua University（清华大学）；Google（谷歌）
6D object pose estimation is a fundamental problem in computer vision. Convolutional Neural Networks (CNNs) have recently proven to be capable of predicting reliable 6D pose estimates even from monocular images. Nonetheless, CNNs are identified as being extremely data-driven, and acquiring adequate annotations is oftentimes very time-consuming and labor intensive. To overcome this shortcoming, we propose the idea of monocular 6D pose estimation by means of self-supervised learning, removing the need for real annotations. After training our proposed network fully supervised with synthetic RGB d…
HierGF: Hierarchical Gaussian Fields via Geometry-perception Message Passing for Sparse-view 3D Reconstruction
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Institute for Math & AI, Wuhan University（武汉大学数学与人工智能研究院）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）
JADE-GS: Joint Allocation of Deblurring Evidence for Event-Assisted 3D Gaussian Splatting
School of Mechatronic Engineering and Automation, Shanghai University（上海大学机电工程与自动化学院）；Shanghai Baoshan Shangda General Intelligent Robotics Research Institute（上海宝山尚大通用智能机器人研究院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）
University of Oxford（牛津大学）；Federal University of Rio Grande（里奥格兰德联邦大学）
We present in this work the first end-to-end deep learning based method that predicts both 3D hand shape and pose from RGB images in the wild. Our network consists of the concatenation of a deep convolutional encoder, and a fixed model-based decoder. Given an input image, and optionally 2D joint detections obtained from an independent CNN, the encoder predicts a set of hand and view parameters. The decoder has two components: A pre-computed articulated mesh deformation hand model that generates a 3D mesh from the hand parameters, and a re-projection module controlled by the view parameters tha…
Improving Neural Indoor Surface Reconstruction with Mask-Guided Adaptive Consistency Constraints
College of Information Engineering, Zhejiang University of Technology（浙江工业大学信息工程学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Tsinghua University（清华大学）；BNRist（北京国家信息科学与技术研究中心）；Google Inc.（谷歌公司）；Technical University of Munich（慕尼黑工业大学）
6D object pose estimation is a fundamental yet challenging problem in computer vision. Convolutional Neural Networks (CNNs) have recently proven to be capable of predicting reliable 6D pose estimates even under monocular settings. Nonetheless, CNNs are identified as being extremely data-driven, and acquiring adequate annotations is oftentimes very time-consuming and labor intensive. To overcome this limitation, we propose a novel monocular 6D pose estimation approach by means of self-supervised learning, removing the need for real annotations. After training our proposed network fully supervis…
Supervised approaches to 3D pose estimation from single images are remarkably effective when labeled data is abundant. However, as the acquisition of ground-truth 3D labels is labor intensive and time consuming, recent attention has shifted towards semi- and weakly-supervised learning. Generating an effective form of supervision with little annotations still poses major challenge in crowded scenes. In this paper we propose to impose multi-view geometrical constraints by means of a weighted differentiable triangulation and use it as a form of self-supervision when no labels are available. We th…
Establishing a correspondence between two non-rigidly deforming shapes is one of the most fundamental problems in visual computing. Existing methods often show weak resilience when presented with challenges innate to real-world data such as noise, outliers, self-occlusion etc. On the other hand, auto-decoders have demonstrated strong expressive power in learning geometrically meaningful latent embeddings. However, their use in \emph{shape analysis} has been limited. In this paper, we introduce an approach based on an auto-decoder framework, that learns a continuous shape-wise deformation field…
Facebook AI Research（Facebook人工智能研究院）；University of Michigan（密歇根大学）；Carnegie Mellon University（卡内基梅隆大学）
We present an approach to infer the 3D shape, texture, and camera pose for an object from a single RGB image, using only category-level image collections with foreground masks as supervision. We represent the shape as an image-conditioned implicit function that transforms the surface of a sphere to that of the predicted mesh, while additionally predicting the corresponding texture. To derive supervisory signal for learning, we enforce that: a) our predictions when rendered should explain the available image evidence, and b) the inferred 3D structure should be geometrically consistent with lear…
DriveSplat: Unified Neural Gaussian Reconstruction for Dynamic Driving Scenes
State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Zhongguancun Academy（中关村学院）；Waytous
Surfsvr: 2D Surface Priors as 3D Geometric Regularizers for Sparse Voxel Reconstruction
HIT (Shenzhen)（哈尔滨工业大学（深圳））；Tsinghua University（清华大学）；NWPU（西北工业大学）；Beijing Institute of Control and Engineering（北京控制与工程研究所）；Technical University of Munich（慕尼黑工业大学）
University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. This is inspired by the observation that view-based surface representations are more effective at modeling high-resolution surface details and texture than their 3D counterparts based on point clouds or voxel occupancy. Specifically, given a 3D shape, we render it from multiple views, and set up a dense correspondence learning task within the contrastive learning framework. As a result, the learned 2D representations are view-invariant and geometrically consistent, leading to better …
KAIST（韩国科学技术院）；DGIST（大邱庆北科学技术院）；Microsoft Research Asia（微软亚洲研究院）
We present an end-to-end joint training framework that explicitly models 6-DoF motion of multiple dynamic objects, ego-motion and depth in a monocular camera setup without supervision. Our technical contributions are three-fold. First, we highlight the fundamental difference between inverse and forward projection while modeling the individual motion of each rigid object, and propose a geometrically correct projection pipeline using a neural forward projection module. Second, we design a unified instance-aware photometric and geometric consistency loss that holistically imposes self-supervisory…
Enhancing Novel View Synthesis from extremely sparse views with SfM-free 3D Gaussian Splatting Framework
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（电子与电气工程系，香港理工大学）；School of Intelligent Systems Engineering, Sun Yat-sen University（智能系统工程学院，中山大学）
Recent conditional image synthesis approaches provide high-quality synthesized images. However, it is still challenging to accurately adjust image contents such as the positions and orientations of objects, and synthesized images often have geometrically invalid contents. To provide users with rich controllability on synthesized images in the aspect of 3D geometry, we propose a novel approach to realistic-looking image synthesis based on a configurable 3D scene layout. Our approach takes a 3D scene with semantic class labels as input and trains a 3D scene painting network that synthesizes colo…
HumanRecon: Neural Reconstruction of Dynamic Human Using Geometric Cues and Physical Priors
Beijing University of Posts and Telecommunications（北京邮电大学）；DJI Technology（大疆创新科技）；Zhejiang University（浙江大学）；Tampere Universities（坦佩雷大学）；University of Adelaide（阿德莱德大学）
Finding semantic correspondences is a challenging problem. With the breakthrough of CNNs stronger features are available for tasks like classification but not specifically for the requirements of semantic matching. In the following we present a weakly supervised metric learning approach which generates stronger features by encoding far more context than previous methods. First, we generate more suitable training data using a geometrically informed correspondence mining method which is less prone to spurious matches and requires only image category labels as supervision. Second, we introduce a …
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Idiap Research Institute（伊迪亚普研究所）；NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（矢量研究所）
Impressive progress in 3D shape extraction led to representations that can capture object geometries with high fidelity. In parallel, primitive-based methods seek to represent objects as semantically consistent part arrangements. However, due to the simplicity of existing primitive representations, these methods fail to accurately reconstruct 3D shapes using a small number of primitives/parts. We address the trade-off between reconstruction quality and number of parts with Neural Parts, a novel 3D primitive representation that defines primitives using an Invertible Neural Network (INN) which i…
Tel Aviv University（特拉维夫大学）；Hebrew University（希伯来大学）；Intel Corporation（英特尔公司）
The process of aligning a pair of shapes is a fundamental operation in computer graphics. Traditional approaches rely heavily on matching corresponding points or features to guide the alignment, a paradigm that falters when significant shape portions are missing. These techniques generally do not incorporate prior knowledge about expected shape characteristics, which can help compensate for any misleading cues left by inaccuracies exhibited in the input shapes. We present an approach based on a deep neural network, leveraging shape datasets to learn a shape-aware prior for source-to-target ali…
