Estimating the pose of a camera with respect to a 3D reconstruction or scene representation is a crucial step for many mixed reality and robotics applications. Given the vast amount of available data nowadays, many applications constrain storage and/or bandwidth to work efficiently. To satisfy these constraints, many applications compress a scene representation by reducing its number of 3D points. While state-of-the-art methods use $K$-cover-based algorithms to compress a scene, they are slow and hard to tune. To enhance speed and facilitate parameter tuning, this work introduces a novel approach that compresses a scene representation by means of a constrained quadratic program (QP). Because this QP resembles a one-class support vector machine, we derive a variant of the sequential minimal optimization to solve it. Our approach uses the points corresponding to the support vectors as the subset of points to represent a scene. We also present an efficient initialization method that allows our method to converge quickly. Our experiments on publicly available datasets show that our approach compresses a scene representation quickly while delivering accurate pose estimates.
Rendering 3D scenes requires access to arbitrary viewpoints from the scene. Storage of such a 3D scene can be done in two ways; (1) storing 2D images taken from the 3D scene that can reconstruct the scene back through interpolations, or (2) storing a representation of the 3D scene itself that already encodes views from all directions. So far, traditional 3D compression methods have focused on the first type of storage and compressed the original 2D images with image compression techniques. With this approach, the user first decodes the stored 2D images and then renders the 3D scene. However, t…
Improving 3D Gaussian Splatting Compression by Scene-Adaptive Lattice Vector Quantization
Department of Electrical & Computer Engineering, McMaster University（麦卡特尼大学电气与计算机工程系）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）
This paper presents an end-to-end neural mapping method for camera localization, dubbed NeuMap, encoding a whole scene into a grid of latent codes, with which a Transformer-based auto-decoder regresses 3D coordinates of query pixels. State-of-the-art feature matching methods require each scene to be stored as a 3D point cloud with per-point features, consuming several gigabytes of storage per scene. While compression is possible, performance drops significantly at high compression rates. Conversely, coordinate regression methods achieve high compression by storing scene information in a neural…
Image-based localization is a core component of many augmented/mixed reality (AR/MR) and autonomous robotic systems. Current localization systems rely on the persistent storage of 3D point clouds of the scene to enable camera pose estimation, but such data reveals potentially sensitive scene information. This gives rise to significant privacy risks, especially as for many applications 3D mapping is a background process that the user might not be fully aware of. We pose the following question: How can we avoid disclosing confidential information about the captured 3D scene, and yet allow reliab…
SparseGF: A Height-Aware Sparse Segmentation Framework with Context Compression for Robust Ground Filtering Across Urban to Natural Scenes
School of Remote Sensing and Geomatics Engineering, Nanjing University of Information Science and Technology（南京信息工程大学遥感与地理信息工程学院）；Tiandu-Nuist Deep Space Exploration Laboratory（天都-南京信息工程大学深空探测实验室）；School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感与信息工程学院）；School of Land Science and Technology, China University of Geosciences（中国地质大学（北京）土地科学与技术学院）；School of Geospatial Artificial Intelligence, East China Normal University（华东师范大学地理信息智能学院）；Department of Geography and Environmental Management, University of Waterloo（滑铁卢大学地理与环境管理系）；Department of Systems Design Engineering, University of Waterloo（滑铁卢大学系统设计工程系）
TinySplat: Feedforward Approach for Generating Compact 3D Scene Representation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室、计算机学院、北京大学）；School of Electronic and Computer Engineering, Peking University（电子与计算机工程学院、北京大学）；Wangxuan Institute of Computer Technology, State Key Laboratory of Multimedia Information Processing, Peking University（计算机技术王轩研究所、多媒体信息处理国家重点实验室、北京大学）
SparseStreet: Sparse Gaussian Splatting for Real-Time Street Scene Simulation
Peking University（北京大学）；Chinese Academy of Sciences（中国科学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Autonomous Driving Development, NIO（蔚来自动驾驶开发）
Shandong University（山东大学）；Peking University（北京大学）；Tencent AI Lab（腾讯人工智能实验室）；Stanford University（斯坦福大学）；vjia（3vjia）
In this work, we tackle the problem of active camera localization, which controls the camera movements actively to achieve an accurate camera pose. The past solutions are mostly based on Markov Localization, which reduces the position-wise camera uncertainty for localization. These approaches localize the camera in the discrete pose space and are agnostic to the localization-driven scene property, which restricts the camera pose accuracy in the coarse scale. We propose to overcome these limitations via a novel active camera localization algorithm, composed of a passive and an active localizati…
Efficient Sparse-to-Dense Visual Localization via Compact Gaussian Scene Representation and Accelerated Dense Pose Estimation
Electronic Information School, Wuhan University（武汉大学电子信息学院）；School of Robotics, Wuhan University（武汉大学机器人学院）；Electronic Information School and the School of Robotics, Wuhan University（武汉大学电子信息学院和机器人学院）
ExGS: Extreme 3D Gaussian Compression with Diffusion Priors
Northwestern Polytechnical University（北western工业大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology（香港科技大学）
ETH Zurich（苏黎世联邦理工学院）；Chalmers University of Technology（查尔姆斯理工大学）；Eigenvision；Ecole des Ponts（巴黎高科路桥学校）；Microsoft（微软）；Czech Technical University in Prague（捷克理工大学）
Camera pose estimation in known scenes is a 3D geometry task recently tackled by multiple learning algorithms. Many regress precise geometric quantities, like poses or 3D points, from an input image. This either fails to generalize to new viewpoints or ties the model parameters to a specific scene. In this paper, we go Back to the Feature: we argue that deep networks should focus on learning robust and invariant visual features, while the geometric estimation should be left to principled algorithms. We introduce PixLoc, a scene-agnostic neural network that estimates an accurate 6-DoF pose from…
Czech Institute of Informatics, Robotics and Cybernetics（捷克信息学、机器人学与控制论研究所）
Visual localization, i.e., the problem of camera pose estimation, is a central component of applications such as autonomous robots and augmented reality systems. A dominant approach in the literature, shown to scale to large scenes and to handle complex illumination and seasonal changes, is based on local features extracted from images. The scene representation is a sparse Structure-from-Motion point cloud that is tied to a specific local feature. Switching to another feature type requires an expensive feature matching step between the database images used to construct the point cloud. In this…
Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes
ROCS（罗克辛研究所）；Balgrist University Hospital（巴尔格里斯大学医院）；University of Zurich（苏黎世大学）；Dept. of Orthopedics（骨科部门）；Computer Vision and Geometry Group（计算机视觉与几何组）；ETH Zurich（苏黎世联邦理工学院）
Reliable-loc: Robust sequential LiDAR global localization in large-scale street scenes based on verifiable cues
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Earth Sciences and Engineering, Hohai University（河海大学地球科学与工程学院）；Hubei Luojia Laboratory（湖北珞珈实验室）
We consider learning based methods for visual localization that do not require the construction of explicit maps in the form of point clouds or voxels. The goal is to learn an implicit representation of the environment at a higher, more abstract level. We propose to use a generative approach based on Generative Query Networks (GQNs, Eslami et al. 2018), asking the following questions: 1) Can GQN capture more complex scenes than those it was originally demonstrated on? 2) Can GQN be used for localization in those scenes? To study this approach we consider procedurally generated Minecraft worlds…
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）；Soochow University（苏州大学）
Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environmen…
ETH Zurich（苏黎世联邦理工学院）；Microsoft（微软）；Chalmers University of Technology（查尔姆斯理工大学）
Localizing an image wrt. a 3D scene model represents a core task for many computer vision applications. An increasing number of real-world applications of visual localization on mobile devices, e.g., Augmented Reality or autonomous robots such as drones or self-driving cars, demand localization approaches to minimize storage and bandwidth requirements. Compressing the 3D models used for localization thus becomes a practical necessity. In this work, we introduce a new hybrid compression algorithm that uses a given memory limit in a more effective way. Rather than treating all 3D points equally,…
Simon Fraser University（西蒙菲莎大学）；Alibaba A.I Labs（阿里巴巴人工智能实验室）
Camera localization aims to estimate 6 DoF camera poses from RGB images. Traditional methods detect and match interest points between a query image and a pre-built 3D model. Recent learning-based approaches encode scene structures into a specific convolutional neural network (CNN) and thus are able to predict dense coordinates from RGB images. However, most of them require re-training or re-adaption for a new scene and have difficulties in handling large-scale scenes due to limited network capacity. We present a new method for scene agnostic camera localization using dense scene matching (DSM)…
Visual (re)localization addresses the problem of estimating the 6-DoF (Degree of Freedom) camera pose of a query image captured in a known scene, which is a key building block of many computer vision and robotics applications. Recent advances in structure-based localization solve this problem by memorizing the mapping from image pixels to scene coordinates with neural networks to build 2D-3D correspondences for camera pose optimization. However, such memorization requires training by amounts of posed images in each scene, which is heavy and inefficient. On the contrary, few-shot images are usu…
Aalto University（阿尔托大学）；Facebook AI Research（Facebook人工智能研究院）；INRIA（法国国家信息与自动化研究所）
Visual localization is critical to many applications in computer vision and robotics. To address single-image RGB localization, state-of-the-art feature-based methods match local descriptors between a query image and a pre-built 3D model. Recently, deep neural networks have been exploited to regress the mapping between raw pixels and 3D coordinates in the scene, and thus the matching is implicitly performed by the forward pass through the network. However, in a large and ambiguous environment, learning such a regression task directly can be difficult for a single network. In this work, we pres…
Mangalore University（芒格洛尔大学）；KVG College of Engineering（KVG工程学院）
In this paper, we propose a skeleton matching based approach which aids in text localization in scene images. The input image is preprocessed and segmented into blocks using connected component analysis. We obtain the skeleton of the segmented block using morphology based approach. The skeletonized images are compared with the trained templates in the database to categorize into text and non-text blocks. Further, the newly designed geometrical rules and morphological operations are employed on the detected text blocks for scene text localization. The experimental results obtained on publicly a…
Technical University of Munich（慕尼黑工业大学）；Chalmers University of Technology（查尔姆斯理工大学）；CIIRC, Czech Technical University in Prague（捷克布拉格捷克理工大学信息、机器人与控制研究所）；Five AI Ltd.（Five AI有限公司）；Google Inc.（谷歌公司）
Long-term camera re-localization is an important task with numerous computer vision and robotics applications. Whilst various outdoor benchmarks exist that target lighting, weather and seasonal changes, far less attention has been paid to appearance changes that occur indoors. This has led to a mismatch between popular indoor benchmarks, which focus on static scenes, and indoor environments that are of interest for many real-world applications. In this paper, we adapt 3RScan - a recently introduced indoor RGB-D dataset designed for object instance re-localization - to create RIO10, a new long-…
