Traditional visual place recognition (VPR) methods generally use frame-based cameras, which is easy to fail due to dramatic illumination changes or fast motions. In this paper, we propose an end-to-end visual place recognition network for event cameras, which can achieve good place recognition performance in challenging environments. The key idea of the proposed algorithm is firstly to characterize the event streams with the EST voxel grid, then extract features using a convolution network, and finally aggregate features using an improved VLAD network to realize end-to-end visual place recognition using event streams. To verify the effectiveness of the proposed algorithm, we compare the proposed method with classical VPR methods on the event-based driving datasets (MVSEC, DDD17) and the synthetic datasets (Oxford RobotCar). Experimental results show that the proposed method can achieve much better performance in challenging scenarios. To our knowledge, this is the first end-to-end event-based VPR method. The accompanying source code is available at https://github.com/kongdelei/Event-VPR.
Traditional visual place recognition (VPR), usually using standard cameras, is easy to fail due to glare or high-speed motion. By contrast, event cameras have the advantages of low latency, high temporal resolution, and high dynamic range, which can deal with the above issues. Nevertheless, event cameras are prone to failure in weakly textured or motionless scenes, while standard cameras can still provide appearance information in this case. Thus, exploiting the complementarity of standard cameras and event cameras can effectively improve the performance of VPR algorithms. In the paper, we pro…
UGNA-VPR: A Novel Training Paradigm for Visual Place Recognition Based on Uncertainty-Guided NeRF Augmentation
College of Intelligence Science and Technology, National University of Defense Technology（国防科技大学智能科学学院）；Faculty of Robot Science and Engineering, Northeastern University（东北大学机器人科学与工程学院）；School of Electronic and Information Engineering, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）电子与信息工程学院）；Institute of Cyber-Systems and Control, Zhejiang University（浙江大学网络系统与控制研究所）；SIASUN Robot& Automation Co., Ltd（新松机器人自动化股份有限公司）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）
University of Essex（埃塞克斯大学）；Queensland University of Technology（昆士兰科技大学）；University of Southampton（南安普顿大学）
Visual place recognition (VPR) is an essential component of robot navigation and localization systems that allows them to identify a place using only image data. VPR is challenging due to the significant changes in a place's appearance under different illumination throughout the day, with seasonal weather and when observed from different viewpoints. Currently, no single VPR technique excels in every environmental condition, each exhibiting unique benefits and shortcomings. As a result, VPR systems combining multiple techniques achieve more reliable VPR performance in changing environments, at …
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences（中国科学院重庆绿色智能技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Central South University（中南大学）；Texas A&M University（德克萨斯A&M大学）
Recently, the methods based on Convolutional Neural Networks (CNNs) have gained popularity in the field of visual place recognition (VPR). In particular, the features from the middle layers of CNNs are more robust to drastic appearance changes than handcrafted features and high-layer features. Unfortunately, the holistic mid-layer features lack robustness to large viewpoint changes. Here we split the holistic mid-layer features into local features, and propose an adaptive dynamic time warping (DTW) algorithm to align local features from the spatial domain while measuring the distance between t…
Through the Lens of Doubt: Robust and Efficient Uncertainty Estimation for Visual Place Recognition
School of Electronics and Computer Science, University of Southampton（索姆塞特大学电子与计算机科学学院）；School of Electronics and Computer Science, Queensland University of Technology（昆士兰科技大学电子与计算机科学学院）；School of Computer Science and Electronic Engineering, University of Essex（埃塞克斯大学计算机科学与电子工程学院）
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所机器人学国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Biorobotics Lab, Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所生物机器人实验室）
Place recognition and loop closure detection are challenging for long-term visual navigation tasks. SeqSLAM is considered to be one of the most successful approaches to achieving long-term localization under varying environmental conditions and changing viewpoints. It depends on a brute-force, time-consuming sequential matching method. We propose MRS-VPR, a multi-resolution, sampling-based place recognition method, which can significantly improve the matching efficiency and accuracy in sequential matching. The novelty of this method lies in the coarse-to-fine searching pipeline and a particle …
In visual place recognition (VPR), map segmentation (MS) is a preprocessing technique used to partition a given view-sequence map into place classes (i.e., map segments) so that each class has good place-specific training images for a visual place classifier (VPC). Existing approaches to MS implicitly/explicitly suppose that map segments have a certain size, or individual map segments are balanced in size. However, recent VPR systems showed that very small important map segments (minimal map segments) often suffice for VPC, and the remaining large unimportant portion of the map should be disca…
HE-VPR: Height Estimation Enabled Aerial Visual Place Recognition Against Scale Variance
Department of Precision Instrument, Tsinghua University（清华大学精密仪器系）；School of Aerospace Engineering, Beijing Institute of Technology（北京理工大学航天工程学院）；School of Instrumentation Science and Opto-electronics Engineering, Beijing Information Science and Technology University（北京信息科技大学仪器科学与光电工程学院）；School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能系统工程学院）
Visual place recognition for aerial imagery: A survey
Center for AI Technology (CAIT)（人工智能技术中心（CAIT））；Skolkovo Institute of Science and Technology (Skoltech)（斯科尔科沃科学技术学院（Skoltech））；Saint Petersburg State University（圣彼得堡国立大学）
Visual Place Recognition (VPR) is often characterized as being able to recognize the same place despite significant changes in appearance and viewpoint. VPR is a key component of Spatial Artificial Intelligence, enabling robotic platforms and intelligent augmentation platforms such as augmented reality devices to perceive and understand the physical world. In this paper, we observe that there are three "drivers" that impose requirements on spatially intelligent agents and thus VPR systems: 1) the particular agent including its sensors and computational resources, 2) the operating environment o…
Indian Institute of Technology (ISM) Dhanbad（印度理工学院（印度矿业学院）丹巴德分校）；Queensland University of Technology (QUT)（昆士兰科技大学）
Visual Place Recognition (VPR) approaches have typically attempted to match places by identifying visual cues, image regions or landmarks that have high ``utility'' in identifying a specific place. But this concept of utility is not singular - rather it can take a range of forms. In this paper, we present a novel approach to deduce two key types of utility for VPR: the utility of visual cues `specific' to an environment, and to a particular place. We employ contrastive learning principles to estimate both the environment- and place-specific utility of Vector of Locally Aggregated Descriptors (…
Adversarial Attacks and Detection in Visual Place Recognition for Safer Robot Navigation
QUT Centre for Robotics（昆士兰科技大学机器人中心）；School of Electrical Engineering and Robotics at the Queensland University of Technology（昆士兰科技大学电气工程与机器人学院）；CIICADA Lab（CIICADA实验室）
Improving Visual Place Recognition Based Robot Navigation By Verifying Localization Estimates
Queensland University of Technology（昆士兰科技大学）；QUT Centre for Robotics（昆士兰科技大学机器人中心）；School of Electrical Engineering and Robotics（电气工程与机器人学院）；Australian National University（澳大利亚国立大学）；College of Engineering, Computing and Cybernetics（工程、计算与网络学院）
Event cameras are bio-inspired sensors capable of providing a continuous stream of events with low latency and high dynamic range. As a single event only carries limited information about the brightness change at a particular pixel, events are commonly accumulated into spatio-temporal windows for further processing. However, the optimal window length varies depending on the scene, camera motion, the task being performed, and other factors. In this research, we develop a novel ensemble-based scheme for combining temporal windows of varying lengths that are processed in parallel. For application…
Signage is everywhere and a robot should be able to take advantage of signs to help it localize (including Visual Place Recognition (VPR)) and map. Robust text detection & recognition in the wild is challenging due to such factors as pose, irregular text, illumination, and occlusion. We propose an end-to-end scene text spotting model that simultaneously outputs the text string and bounding boxes. This model is more suitable for VPR. Our central contribution is introducing utilizing an end-to-end scene text spotting framework to adequately capture the irregular and occluded text regions in diff…
The University of Sheffield（谢菲尔德大学）；Macquarie University（麦考瑞大学）；University of Sussex（萨塞克斯大学）；Institute for Neuroinformatics（神经信息学研究所）
Recognising previously visited locations is an important, but unsolved, task in autonomous navigation. Current visual place recognition (VPR) benchmarks typically challenge models to recover the position of a query image (or images) from sequential datasets that include both spatial and temporal components. Recently, Echo State Network (ESN) varieties have proven particularly powerful at solving machine learning tasks that require spatio-temporal modelling. These networks are simple, yet powerful neural architectures that--exhibiting memory over multiple time-scales and non-linear high-dimensi…
CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Landmark-based robot self-localization has recently garnered interest as a highly-compressive domain-invariant approach for performing visual place recognition (VPR) across domains (e.g., time of day, weather, and season). However, landmark-based self-localization can be an ill-posed problem for a passive observer (e.g., manual robot control), as many viewpoints may not provide an effective landmark view. In this study, we consider an active self-localization task by an active observer and present a novel reinforcement learning (RL)-based next-best-view (NBV) planner. Our contributions are as …
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology of China（中国科学技术大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Beijing University of Posts and Telecommunications（北京邮电大学）
Visual Place Recognition (VPR) is an important component in both computer vision and robotics applications, thanks to its ability to determine whether a place has been visited and where specifically. A major challenge in VPR is to handle changes of environmental conditions including weather, season and illumination. Most VPR methods try to improve the place recognition performance by ignoring the environmental factors, leading to decreased accuracy decreases when environmental conditions change significantly, such as day versus night. To this end, we propose an end-to-end conditional visual pl…
Spiking neural networks (SNNs) offer both compelling potential advantages, including energy efficiency and low latencies and challenges including the non-differentiable nature of event spikes. Much of the initial research in this area has converted deep neural networks to equivalent SNNs, but this conversion approach potentially negates some of the advantages of SNN-based approaches developed from scratch. One promising area for high-performance SNNs is template matching and image recognition. This research introduces the first high-performance SNN for the Visual Place Recognition (VPR) task: …
Nanyang Technological University（南洋理工大学）；Beijing Institute of Technology（北京理工大学）
Large-scale visual place recognition (VPR) is inherently challenging because not all visual cues in the image are beneficial to the task. In order to highlight the task-relevant visual cues in the feature embedding, the existing attention mechanisms are either based on artificial rules or trained in a thorough data-driven manner. To fill the gap between the two types, we propose a novel Semantic Reinforced Attention Learning Network (SRALNet), in which the inferred attention can benefit from both semantic priors and data-driven fine-tuning. The contribution lies in two-folds. (1) To suppress m…
University of Melbourne（墨尔本大学）；Queensland University of Technology（昆士兰科技大学）
A key challenge in visual place recognition (VPR) is recognizing places despite drastic visual appearance changes due to factors such as time of day, season, weather or lighting conditions. Numerous approaches based on deep-learnt image descriptors, sequence matching, domain translation, and probabilistic localization have had success in addressing this challenge, but most rely on the availability of carefully curated representative reference images of the possible places. In this paper, we propose a novel approach, dubbed Bayesian Selective Fusion, for actively selecting and fusing informativ…
SPOT: Point Cloud Based Stereo Visual Place Recognition for Similar and Opposing Viewpoints
Department of Robotics, University of Michigan（机器人学系，密歇根大学）；Department of Robotics and the Department of Mechanical Engineering, University of Michigan（机器人学系和机械工程系，密歇根大学）
Human visual scene understanding is so remarkable that we are able to recognize a revisited place when entering it from the opposite direction it was first visited, even in the presence of extreme variations in appearance. This capability is especially apparent during driving: a human driver can recognize where they are when travelling in the reverse direction along a route for the first time, without having to turn back and look. The difficulty of this problem exceeds any addressed in past appearance- and viewpoint-invariant visual place recognition (VPR) research, in part because large parts…
Towards Lifelong Aerial Autonomy: Geometric Memory Management for Continual Visual Place Recognition in Dynamic Environments
Department of Precision Instrument, Tsinghua University（清华大学精密仪器系）；College of Instrument Science and Opto-electronics Engineering, Beijing Information Science and Technology University（北京信息科技大学仪器科学与光电工程学院）；Key Laboratory of Complex System Intelligent Control and Decision Making, Beijing Institute of Technology（北京理工大学复杂系统智能控制与决策重点实验室）；School of Aerospace Engineering, Beijing Institute of Technology（北京理工大学宇航学院）
VIPeR: Visual Incremental Place Recognition with Adaptive Mining and Continual Learning
School of Computer Science, Hangzhou Dianzi University（杭州电子科技大学计算机学院）；High-speed Aerodynamics Institute, CARDC（中国空气动力研究与发展中心高速空气动力研究所）；State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）；Stevens Institute of Technology（史蒂文斯理工学院）
We present the ALTO dataset, a vision-focused dataset for the development and benchmarking of Visual Place Recognition and Localization methods for Unmanned Aerial Vehicles. The dataset is composed of two long (approximately 150km and 260km) trajectories flown by a helicopter over Ohio and Pennsylvania, and it includes high precision GPS-INS ground truth location data, high precision accelerometer readings, laser altimeter readings, and RGB downward facing camera imagery. In addition, we provide reference imagery over the flight paths, which makes this dataset suitable for VPR benchmarking and…
Event cameras continue to attract interest due to desirable characteristics such as high dynamic range, low latency, virtually no motion blur, and high energy efficiency. One of the potential applications that would benefit from these characteristics lies in visual place recognition for robot localization, i.e. matching a query observation to the corresponding reference place in the database. In this letter, we explore the distinctiveness of event streams from a small subset of pixels (in the tens or hundreds). We demonstrate that the absolute difference in the number of events at those pixel …
