Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions
ETH Zürich（苏黎世联邦理工学院）；University of Oxford（牛津大学）；Chalmers University of Technology（查尔姆斯理工大学）；Tokyo Institute of Technology（东京工业大学）；Technical University of Denmark（丹麦技术大学）；Microsoft（微软）；Inria（法国国家信息与自动化研究所）；CTU in Prague（布拉格捷克技术大学）；Centre for Mathematical Sciences, Lund University（隆德大学数学科学中心）；ENS/INRIA/CNRS UMR 8548, PSL Research University（巴黎高等师范学院/法国国家信息与自动化研究所/法国国家科学研究中心联合实验室（编号8548），巴黎高等师范学院 - 巴黎科学与人文学院（PSL Research University））；CIIRC - Czech Institute of Informatics, Robotics, and Cybernetics, Czech Technical University in Prague（布拉格捷克技术大学捷克信息学、机器人学与控制论研究所（CIIRC））；WILLOW project（WILLOW项目）
Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, Fredrik Kahl, Tomas Pajdla
Visual localization enables autonomous vehicles to navigate in their surroundings and augmented reality applications to link virtual to real worlds. Practical visual localization approaches need to be robust to a wide variety of viewing condition, including day-night changes, as well as weather and seasonal variations, while providing highly accurate 6 degree-of-freedom (6DOF) camera pose estimates. In this paper, we introduce the first benchmark datasets specifically designed for analyzing the impact of such factors on visual localization. Using carefully created ground truth poses for query images taken under a wide variety of conditions, we evaluate the impact of various factors on 6DOF camera pose estimation accuracy through extensive experiments with state-of-the-art localization approaches. Based on our results, we draw conclusions about the difficulty of different conditions, showing that long-term localization is far from solved, and propose promising avenues for future work, including sequence-based localization approaches and the need for better local features. Our benchmark is available at visuallocalization.net.
We present the Brown Pedestrian Odometry Dataset (BPOD) for benchmarking visual odometry algorithms in head-mounted pedestrian settings. This dataset was captured using synchronized global and rolling shutter stereo cameras in 12 diverse indoor and outdoor locations on Brown University's campus. Compared to existing datasets, BPOD contains more image blur and self-rotation, which are common in pedestrian odometry but rare elsewhere. Ground-truth trajectories are generated from stick-on markers placed along the pedestrian's path, and the pedestrian's position is documented using a third-person …
Benchmarking foundation models for hyperspectral image classification: Application to cereal crop type mapping
Analytics Lab (A-Lab), UM6P, Rabat 11103, Morocco（安利实验室（A-Lab））；Center for Remote Sensing Applications (CRSA), UM6P, Ben Guerir 43150, Morocco（远程传感应用中心（CRSA））；Friedrich Schiller University Jena, Department of Geography, Jena 07743, Germany（耶拿弗里德里希·施勒格尔大学）；Dept. of Cartography（制图部门）
Cryo-Bench: Benchmarking Foundation Models for Cryosphere Mapping
Center for Sustainability and the Global Environment (SAGE), University of Wisconsin–Madison（可持续性与全球环境中心（SAGE），威斯康星大学麦迪逊分校）；Portsmouth AI and Data Science Centre (PAIDS), School of Computing, University of Portsmouth（波特茅斯人工智能与数据科学中心（PAIDS），计算学院，波特茅斯大学）；ESA, ESRIN, φ \varphi -lab, Frascati（欧洲航天局（ESA），欧洲空间研究中心（ESRIN），φ实验室，弗拉斯卡蒂）
Assessing the value of Geo-Foundational Models for Flood Inundation Mapping: Benchmarking models for Sentinel-1, Sentinel-2, and Planetscope for end-users
School of Computing, University of Portsmouth（朴茨茅斯大学计算学院）
Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement
Tsinghua University（清华大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Migu Beijing Research Institute（咪咕北京研究院）；School of Architecture, Tsinghua University（清华大学建筑学院）；Department of Automation, Tsinghua University（清华大学自动化系）；BNRist, Tsinghua University（清华大学北京信息科学与技术国家研究中心）
Commercial visual-inertial odometry (VIO) systems have been gaining attention as cost-effective, off-the-shelf six degrees of freedom (6-DoF) ego-motion tracking methods for estimating accurate and consistent camera pose data, in addition to their ability to operate without external localization from motion capture or global positioning systems. It is unclear from existing results, however, which commercial VIO platforms are the most stable, consistent, and accurate in terms of state estimation for indoor and outdoor robotic applications. We assess four popular proprietary VIO systems (Apple A…
Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance
FPT Software AI Center（FPT软件AI中心）；TU Wien（维也纳技术大学）；AIT GmbH（AIT有限公司）；Imperial College London（伦敦帝国学院）；Physical Intelligence（物理智能公司）；University of Arkansas（阿肯色大学）；Ton Duc Thang University（同塔大学）；University of Liverpool（利物浦大学）
NeuroLoc: Encoding Navigation Cells for 6-DOF Camera Localization
Software Engineering Institute, East China Normal University（华东师范大学软件工程学院）；School of Geospatial Information, Information Engineering University（信息工程大学地理信息学院）；University of Shanghai for Science and Technology（上海科技大学）；Fujian (Quanzhou) Institute of Advanced Manufacturing Technology, China（福建（泉州）先进制造技术研究院）
SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarking
Machine Intelligence Group, Department of CS&IS, Birla Institute of Technology and Science, Pilani – Hyderabad Campus（人工智能组，计算机科学与信息系，比拉理工学院和科学学院，比拉-海得拉巴校区）
China Unicom（中国联通）；AI Innovation and Application Center（人工智能创新应用中心）
We propose a novel visual localization network when outside environment has changed such as different illumination, weather and season. The visual localization network is composed of a feature extraction network and pose regression network. The feature extraction network is made up of an encoder network based on the Generative Adversarial Network CycleGAN, which can capture intrinsic appearance-invariant feature maps from unpaired samples of different weathers and seasons. With such an invariant feature, we use a 6-DoF pose regression network to tackle long-term visual localization in the pres…
Fraunhofer FKIE（弗劳恩霍夫通信、信息处理与人机工程研究所）；Technical University of Munich（慕尼黑工业大学）
Visual Place Recognition and Visual Localization are essential components in navigation and mapping for autonomous vehicles especially in GNSS-denied navigation scenarios. Recent work has focused on ground or close to ground applications such as self-driving cars or indoor-scenarios and low-altitude drone flights. However, applications such as Urban Air Mobility require operations in large-scale outdoor environments at medium to high altitudes. We present a new dataset named VPAIR. The dataset was recorded on board a light aircraft flying at an altitude of more than 300 meters above ground cap…
Although a wide variety of deep neural networks for robust Visual Odometry (VO) can be found in the literature, they are still unable to solve the drift problem in long-term robot navigation. Thus, this paper aims to propose novel deep end-to-end networks for long-term 6-DoF VO task. It mainly fuses relative and global networks based on Recurrent Convolutional Neural Networks (RCNNs) to improve the monocular localization accuracy. Indeed, the relative sub-networks are implemented to smooth the VO trajectory, while global subnetworks are designed to avoid drift problem. All the parameters are j…
Aalto University（阿尔托大学）；Tampere University of Technology（坦佩雷理工大学）
The lack of realistic and open benchmarking datasets for pedestrian visual-inertial odometry has made it hard to pinpoint differences in published methods. Existing datasets either lack a full six degree-of-freedom ground-truth or are limited to small spaces with optical tracking systems. We take advantage of advances in pure inertial navigation, and develop a set of versatile and challenging real-world computer vision benchmark sets for visual-inertial odometry. For this purpose, we have built a test rig equipped with an iPhone, a Google Pixel Android phone, and a Google Tango device. We prov…
We present a robust and real-time monocular six degree of freedom relocalization system. Our system trains a convolutional neural network to regress the 6-DOF camera pose from a single RGB image in an end-to-end manner with no need of additional engineering or graph optimisation. The algorithm can operate indoors and outdoors in real time, taking 5ms per frame to compute. It obtains approximately 2m and 6 degree accuracy for large scale outdoor scenes and 0.5m and 10 degree accuracy indoors. This is achieved using an efficient 23 layer deep convnet, demonstrating that convnets can be used to s…
In this paper, we present a visual localization pipeline, namely MegLoc, for robust and accurate 6-DoF pose estimation under varying scenarios, including indoor and outdoor scenes, different time across a day, different seasons across a year, and even across years. MegLoc achieves state-of-the-art results on a range of challenging datasets, including winning the Outdoor and Indoor Visual Localization Challenge of ICCV 2021 Workshop on Long-term Visual Localization under Changing Conditions, as well as the Re-localization Challenge for Autonomous Driving of ICCV 2021 Workshop on Map-based Local…
Dansk Drone Kompagni ApS（丹麦无人机公司）；Technical University of Denmark（丹麦技术大学）；Universidad de Zaragoza（萨拉戈萨大学）
Place recognition and visual localization are particularly challenging in wide baseline configurations. In this paper, we contribute with the \emph{Danish Airs and Grounds} (DAG) dataset, a large collection of street-level and aerial images targeting such cases. Its main challenge lies in the extreme viewing-angle difference between query and reference images with consequent changes in illumination and perspective. The dataset is larger and more diverse than current publicly available data, including more than 50 km of road in urban, suburban and rural areas. All images are associated with acc…
Chalmers University of Technology（查尔姆斯理工大学）；Facebook（脸书）；Czech Technical University in Prague（布拉格捷克理工大学）
Visual localization is the problem of estimating the position and orientation from which a given image (or a sequence of images) is taken in a known scene. It is an important part of a wide range of computer vision and robotics applications, from self-driving cars to augmented/virtual reality systems. Visual localization techniques should work reliably and robustly under a wide range of conditions, including seasonal, weather, illumination and man-made changes. Recent benchmarking efforts model this by providing images under different conditions, and the community has made rapid progress on th…
Hong Kong University of Science and Technology（香港科技大学）
The monocular visual-inertial system (VINS), which consists one camera and one low-cost inertial measurement unit (IMU), is a popular approach to achieve accurate 6-DOF state estimation. However, such locally accurate visual-inertial odometry is prone to drift and cannot provide absolute pose estimation. Leveraging history information to relocalize and correct drift has become a hot topic. In this paper, we propose a monocular visual-inertial SLAM system, which can relocalize camera and get the absolute pose in a previous-built map. Then 4-DOF pose graph optimization is performed to correct dr…
We present a real-time approach for image-based localization within large scenes that have been reconstructed offline using structure from motion (Sfm). From monocular video, our method continuously computes a precise 6-DOF camera pose, by efficiently tracking natural features and matching them to 3D points in the Sfm point cloud. Our main contribution lies in efficiently interleaving a fast keypoint tracker that uses inexpensive binary feature descriptors with a new approach for direct 2D-to-3D matching. The 2D-to-3D matching avoids the need for online extraction of scale-invariant features. …
Queensland University of Technology（昆士兰科技大学）；Singapore University of Technology and Design（新加坡科技设计大学）；University of Liverpool（利物浦大学）
Visual localization has become a key enabling component of many place recognition and SLAM systems. Contemporary research has primarily focused on improving accuracy and precision-recall type metrics, with relatively little attention paid to a system's absolute storage scaling characteristics, its flexibility to adapt to available computational resources, and its longevity with respect to easily incorporating newly learned or hand-crafted image descriptors. Most significantly, improvement in one of these aspects typically comes at the cost of others: for example, a snapshot-based system that a…
The University of Western Australia（西澳大学）；Murdoch University（莫道克大学）
This paper proposes a novel concept to directly match feature descriptors extracted from 2D images with feature descriptors extracted from 3D point clouds. We use this concept to directly localize images in a 3D point cloud. We generate a dataset of matching 2D and 3D points and their corresponding feature descriptors, which is used to learn a Descriptor-Matcher classifier. To localize the pose of an image at test time, we extract keypoints and feature descriptors from the query image. The trained Descriptor-Matcher is then used to match the features from the image and the point cloud. The loc…
Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）；Capgemini（凯捷咨询）；Carnegie Mellon University（卡内基梅隆大学）；University of Michigan（密歇根大学）
A potential Mars Sample Return (MSR) architecture is being jointly studied by NASA and ESA. As currently envisioned, the MSR campaign consists of a series of 3 missions: sample cache, fetch and return to Earth. In this paper, we focus on the fetch part of the MSR, and more specifically the problem of autonomously detecting and localizing sample tubes deposited on the Martian surface. Towards this end, we study two machine-vision based approaches: First, a geometry-driven approach based on template matching that uses hard-coded filters and a 3D shape model of the tube; and second, a data-driven…
Visual Place Recognition (VPR) is a crucial component of 6-DoF localization, visual SLAM and structure-from-motion pipelines, tasked to generate an initial list of place match hypotheses by matching global place descriptors. However, commonly-used CNN-based methods either process multiple image resolutions after training or use a single resolution and limit multi-scale feature extraction to the last convolutional layer during training. In this paper, we augment NetVLAD representation learning with low-resolution image pyramid encoding which leads to richer place representations. The resultant …
RVN-Bench: A Benchmark for Reactive Visual Navigation
Department of Electrical and Computer Engineering, Seoul National University (SNU) and Automation and Systems Research Institute (ASRI) and Sequor Robotics Inc.（电气与计算机工程系，首尔国立大学（SNU）和自动化与系统研究所（ASRI）以及Sequor机器人公司）；Department of Electrical and Computer Engineering, Seoul National University (SNU) and Automation and Systems Research Institute (ASRI)（电气与计算机工程系，首尔国立大学（SNU）和自动化与系统研究所（ASRI））；Interdisciplinary Program in Artificial Intelligence, Seoul National University (SNU) and Automation and Systems Research Institute (ASRI) and Sequor Robotics Inc.（人工智能跨学科项目，首尔国立大学（SNU）和自动化与系统研究所（ASRI）以及Sequor机器人公司）；Sequor Robotics Inc.（Sequor机器人公司）
Visual (re)localization addresses the problem of estimating the 6-DoF (Degree of Freedom) camera pose of a query image captured in a known scene, which is a key building block of many computer vision and robotics applications. Recent advances in structure-based localization solve this problem by memorizing the mapping from image pixels to scene coordinates with neural networks to build 2D-3D correspondences for camera pose optimization. However, such memorization requires training by amounts of posed images in each scene, which is heavy and inefficient. On the contrary, few-shot images are usu…
Visual localization is one of the most important components for robotics and autonomous driving. Recently, inspiring results have been shown with CNN-based methods which provide a direct formulation to end-to-end regress 6-DoF absolute pose. Additional information like geometric or semantic constraints is generally introduced to improve performance. Especially, the latter can aggregate high-level semantic information into localization task, but it usually requires enormous manual annotations. To this end, we propose a novel auxiliary learning strategy for camera localization by introducing sce…
The Graduate Center, City University of New York（纽约城市大学研究生中心）；Tufts University（塔夫茨大学）
Images captured in real-world applications in remote sensing, image or video retrieval, and outdoor surveillance suffer degraded quality introduced by poor weather conditions. Conditions such as rain and mist, introduce artifacts that make visual analysis challenging and limit the performance of high-level computer vision methods. For time-critical applications where a rapid response is necessary, it becomes crucial to develop algorithms that automatically remove rain, without diminishing the quality of the image contents. This article aims to develop a novel quaternion multi-stage multiscale …
University of Zurich（苏黎世大学）；Czech Technical University in Prague（捷克布拉格技术大学）
Visual Localization is one of the key enabling technologies for autonomous driving and augmented reality. High quality datasets with accurate 6 Degree-of-Freedom (DoF) reference poses are the foundation for benchmarking and improving existing methods. Traditionally, reference poses have been obtained via Structure-from-Motion (SfM). However, SfM itself relies on local features which are prone to fail when images were taken under different conditions, e.g., day/ night changes. At the same time, manually annotating feature correspondences is not scalable and potentially inaccurate. In this work,…
ESVO2: Direct Visual-Inertial Odometry with Stereo Event Cameras
School of Robotics, Hunan University（湖南大学机器人学院）；Neuromorphic Automation and Intelligence Lab (NAIL)（神经形态自动化与智能实验室）；Department of Electronic and Computer Engineering（电子与计算机工程系）；Hong Kong University of Science and Technology（香港科学与技术大学）；TU Berlin（柏林工业大学）；Science of Intelligence Excellence Cluster（智能科学卓越集群）；Robotics Institute Germany（德国机器人研究所）；Einstein Center Digital Future（爱因斯坦中心数字未来）
Semantic understanding and localization are fundamental enablers of robot autonomy that have for the most part been tackled as disjoint problems. While deep learning has enabled recent breakthroughs across a wide spectrum of scene understanding tasks, its applicability to state estimation tasks has been limited due to the direct formulation that renders it incapable of encoding scene-specific constrains. In this work, we propose the VLocNet++ architecture that employs a multitask learning approach to exploit the inter-task relationship between learning semantics, regressing 6-DoF global pose a…
