Visual localization, i.e., camera pose estimation in a known scene, is a core component of technologies such as autonomous driving and augmented reality. State-of-the-art localization approaches often rely on image retrieval techniques for one of two tasks: (1) provide an approximate pose estimate or (2) determine which parts of the scene are potentially visible in a given query image. It is common practice to use state-of-the-art image retrieval algorithms for these tasks. These algorithms are often trained for the goal of retrieving the same landmark under a large range of viewpoint changes. However, robustness to viewpoint changes is not necessarily desirable in the context of visual localization. This paper focuses on understanding the role of image retrieval for multiple visual localization tasks. We introduce a benchmark setup and compare state-of-the-art retrieval representations on multiple datasets. We show that retrieval performance on classical landmark retrieval/recognition tasks correlates only for some but not all tasks to localization performance. This indicates a need for retrieval approaches specifically designed for localization tasks. Our benchmark and evaluation protocols are available at https://github.com/naver/kapture-localization.
CTU in Prague（布拉格捷克理工大学）；Inria Rennes（雷恩法国国家信息与自动化研究所）
In this paper we address issues with image retrieval benchmarking on standard and popular Oxford 5k and Paris 6k datasets. In particular, annotation errors, the size of the dataset, and the level of challenge are addressed: new annotation for both datasets is created with an extra attention to the reliability of the ground truth. Three new protocols of varying difficulty are introduced. The protocols allow fair comparison between different methods, including those using a dataset pre-processing stage. For each dataset, 15 new challenging queries are introduced. Finally, a new set of 1M hard, s…
NAVER LABS Europe（NAVER LABS欧洲）；NAVER LABS（NAVER实验室）；Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（布拉格捷克理工大学捷克信息学、机器人与控制论研究所）
Visual localization, i.e., camera pose estimation in a known scene, is a core component of technologies such as autonomous driving and augmented reality. State-of-the-art localization approaches often rely on image retrieval techniques for one of two purposes: (1) provide an approximate pose estimate or (2) determine which parts of the scene are potentially visible in a given query image. It is common practice to use state-of-the-art image retrieval algorithms for both of them. These algorithms are often trained for the goal of retrieving the same landmark under a large range of viewpoint chan…
Outdoor visual localization is a crucial component to many computer vision systems. We propose an approach to localization from images that is designed to explicitly handle the strong variations in appearance happening between daytime and nighttime. As revealed by recent long-term localization benchmarks, both traditional feature-based and retrieval-based approaches still struggle to handle such changes. Our novel localization method combines a state-of-the-art image retrieval architecture with condition-specific sub-networks allowing the computation of global image descriptors that are explic…
Benchmarking Composed Image Retrieval for Applied Earth Observation
organization= Visual Recognition Group, Department of Cybernetics, Czech Technical University in Prague , country= Czechia；organization= Remote Sensing Laboratory, School of Rural, Surveying；Geoinformatics Engineering, National Technical University of Athens , country= Greece；organization= Institute of Informatics \& Telecommunications, National Centre for Scientific Research ``Demokritos'' , country= Greece；organization= Department of Informatics；Kapodistrian University of Athens , country= Greece
Visual localization tackles the challenge of estimating the camera pose from images by using correspondence analysis between query images and a map. This task is computation and data intensive which poses challenges on thorough evaluation of methods on various datasets. However, in order to further advance in the field, we claim that robust visual localization algorithms should be evaluated on multiple datasets covering a broad domain variety. To facilitate this, we introduce kapture, a new, flexible, unified data format and toolbox for visual localization and structure-from-motion (SFM). It e…
We introduce a novel deep neural network architecture that links visual regions to corresponding textual segments including phrases and words. To accomplish this task, our architecture makes use of the rich semantic information available in a joint embedding space of multi-modal data. From this joint embedding space, we extract the associative localization maps that develop naturally, without explicitly providing supervision during training for the localization task. The joint space is learned using a bidirectional ranking objective that is optimized using a $N$-Pair loss formulation. This tra…
Fine-grained image hashing is a challenging problem due to the difficulties of discriminative region localization and hash code generation. Most existing deep hashing approaches solve the two tasks independently. While these two tasks are correlated and can reinforce each other. In this paper, we propose a deep fine-grained hashing to simultaneously localize the discriminative regions and generate the efficient binary codes. The proposed approach consists of a region localization module and a hash coding module. The region localization module aims to provide informative regions to the hash cod…
This paper addresses the problem of vehicle-mounted camera localization by matching a ground-level image with an overhead-view satellite map. Existing methods often treat this problem as cross-view image retrieval, and use learned deep features to match the ground-level query image to a partition (eg, a small patch) of the satellite map. By these methods, the localization accuracy is limited by the partitioning density of the satellite map (often in the order of tens meters). Departing from the conventional wisdom of image retrieval, this paper presents a novel solution that can achieve highly…
ETH Zürich（苏黎世联邦理工学院）；Chalmers University of Technology（查尔姆斯理工大学）；Microsoft（微软）；KU Leuven（鲁汶大学）
Visual localization is a key step in many robotics pipelines, allowing the robot to (approximately) determine its position and orientation in the world. An efficient and scalable approach to visual localization is to use image retrieval techniques. These approaches identify the image most similar to a query photo in a database of geo-tagged images and approximate the query's pose via the pose of the retrieved database image. However, image retrieval across drastically different illumination conditions, e.g. day and night, is still a problem with unsatisfactory results, even in this age of powe…
This work introduces a simulator-based benchmark for visual localization in the autonomous navigation context. The dynamic benchmark enables investigation of how variables such as the time of day, weather, and camera perspective affect the navigation performance of autonomous agents that utilize visual localization for closed-loop control. The experimental part of the paper studies the effects of four such variables by evaluating state-of-the-art visual localization methods as part of the motion planning module of an autonomous navigation stack. The results show major variation in the suitabil…
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Beijing Key Laboratory of Advanced Information Science and Network Technology（北京先进信息科学与网络技术重点实验室）；State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University（北京交通大学轨道交通控制与安全国家重点实验室）；Centre for Quantum Computation and Intelligent Systems, University of Technology at Sydney（悉尼科技大学量子计算与智能系统中心）
This paper proposes a new evaluation protocol for cross-media retrieval which better fits the real-word applications. Both image-text and text-image retrieval modes are considered. Traditionally, class labels in the training and testing sets are identical. That is, it is usually assumed that the query falls into some pre-defined classes. However, in practice, the content of a query image/text may vary extensively, and the retrieval system does not necessarily know in advance the class label of a query. Considering the inconsistency between the real-world applications and laboratory assumptions…
Università degli Studi di Milano - Bicocca（米兰比可卡大学）
In this report we present the work performed in order to build a dataset for benchmarking vision-based localization at intersections, i.e., a set of stereo video sequences taken from a road vehicle that is approaching an intersection, altogether with a reliable measure of the observer position. This report is meant to complement our paper "Vision-Based Localization at Intersections using Digital Maps" submitted to ICRA2019. It complements the paper because the paper uses the dataset, but it had no space for describing the work done to obtain it. Moreover, the dataset is of interest for all tho…
University of Chinese Academy of Sciences（中国科学院大学）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）
Unsupervised anomaly detection and localization, as of one the most practical and challenging problems in computer vision, has received great attention in recent years. From the time the MVTec AD dataset was proposed to the present, new research methods that are constantly being proposed push its precision to saturation. It is the time to conduct a comprehensive comparison of existing methods to inspire further research. This paper extensively compares 13 papers in terms of the performance in unsupervised anomaly detection and localization tasks, and adds a comparison of inference efficiency p…
The Graduate Center and City College, CUNY（纽约市立大学研究生中心与城市学院）
Indoor localization has many applications, such as commercial Location Based Services (LBS), robotic navigation, and assistive navigation for the blind. This paper formulates the indoor localization problem into a multimedia retrieving problem by modeling visual landmarks with a panoramic image feature, and calculating a user's location via GPU- accelerated parallel retrieving algorithm. To solve the scene similarity problem, we apply a multi-images based retrieval strategy and a 2D aggregation method to estimate the final retrieval location. Experiments on a campus building real data demonstr…
ETH Zürich（苏黎世联邦理工学院）；University of Oxford（牛津大学）；Chalmers University of Technology（查尔姆斯理工大学）；Tokyo Institute of Technology（东京工业大学）；Technical University of Denmark（丹麦技术大学）；Microsoft（微软）；Inria（法国国家信息与自动化研究所）；CTU in Prague（布拉格捷克技术大学）；Centre for Mathematical Sciences, Lund University（隆德大学数学科学中心）；ENS/INRIA/CNRS UMR 8548, PSL Research University（巴黎高等师范学院/法国国家信息与自动化研究所/法国国家科学研究中心联合实验室（编号8548），巴黎高等师范学院 - 巴黎科学与人文学院（PSL Research University））；CIIRC - Czech Institute of Informatics, Robotics, and Cybernetics, Czech Technical University in Prague（布拉格捷克技术大学捷克信息学、机器人学与控制论研究所（CIIRC））；WILLOW project（WILLOW项目）
Visual localization enables autonomous vehicles to navigate in their surroundings and augmented reality applications to link virtual to real worlds. Practical visual localization approaches need to be robust to a wide variety of viewing condition, including day-night changes, as well as weather and seasonal variations, while providing highly accurate 6 degree-of-freedom (6DOF) camera pose estimates. In this paper, we introduce the first benchmark datasets specifically designed for analyzing the impact of such factors on visual localization. Using carefully created ground truth poses for query …
Image features for retrieval-based localization must be invariant to dynamic objects (e.g. cars) as well as seasonal and daytime changes. Such invariances are, up to some extent, learnable with existing methods using triplet-like losses, given a large number of diverse training images. However, due to the high algorithmic training complexity, there exists insufficient comparison between different loss functions on large datasets. In this paper, we train and evaluate several localization methods on three different benchmark datasets, including Oxford RobotCar with over one million images. This …
Beijing University of Posts and Telecommunications（北京邮电大学）；Tencent PCG（腾讯平台与内容事业群）；Tianjin University（天津大学）
The current state-of-the-art methods for video corpus moment retrieval (VCMR) often use similarity-based feature alignment approach for the sake of convenience and speed. However, late fusion methods like cosine similarity alignment are unable to make full use of the information from both query texts and videos. In this paper, we combine feature alignment with feature fusion to promote the performance on VCMR.
SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent Yuanbao（腾讯元宝）；Tsinghua University（清华大学）；ARC Lab, Tencent（腾讯ARC实验室）；The University of Hong Kong（香港大学）
GTPred: Benchmarking MLLMs for Interpretable Geo-localization and Time-of-capture Prediction
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（上海交通大学图像通信与信息处理研究院）；School of Humanities, Shanghai Jiao Tong University（上海交通大学人文学院）；Shanghai AI Laboratory（上海人工智能实验室）
Accurate visual re-localization is very critical to many artificial intelligence applications, such as augmented reality, virtual reality, robotics and autonomous driving. To accomplish this task, we propose an integrated visual re-localization method called RLOCS by combining image retrieval, semantic consistency and geometry verification to achieve accurate estimations. The localization pipeline is designed as a coarse-to-fine paradigm. In the retrieval part, we cascade the architecture of ResNet101-GeM-ArcFace and employ DBSCAN followed by spatial verification to obtain a better initial coa…
MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents
INSAIT；Lanzhou University（兰州大学）；King Abdullah University of Science and Technology（国王阿卜杜勒-阿齐兹科学与技术大学）；Heriot-Watt University（赫瑞-沃德大学）；Monash University（莫纳什大学）；University College Dublin（都柏林大学学院）
ETH Zurich（苏黎世联邦理工学院）；Microsoft（微软）；Lund University（隆德大学）
Localization and mapping is the foundational technology for augmented reality (AR) that enables sharing and persistence of digital content in the real world. While significant progress has been made, researchers are still mostly driven by unrealistic benchmarks not representative of real-world AR scenarios. These benchmarks are often based on small-scale datasets with low scene diversity, captured from stationary cameras, and lack other sensor inputs like inertial, radio, or depth data. Furthermore, their ground-truth (GT) accuracy is mostly insufficient to satisfy AR requirements. To close th…
Shanghai Jiao Tong University（上海交通大学）；University of Cambridge（剑桥大学）
Visual localization is a crucial component in the application of mobile robot and autonomous driving. Image retrieval is an efficient and effective technique in image-based localization methods. Due to the drastic variability of environmental conditions, e.g. illumination, seasonal and weather changes, retrieval-based visual localization is severely affected and becomes a challenging problem. In this work, a general architecture is first formulated probabilistically to extract domain invariant feature through multi-domain image translation. And then a novel gradient-weighted similarity activat…
Shanghai Jiao Tong University（上海交通大学）；State Key Laboratory of Robotics and System (HIT)（机器人技术与系统国家重点实验室（哈工大））；Chinese University of Hong Kong（香港中文大学）；University of the West of England（西英格兰大学）
Visual localization is a crucial problem in mobile robotics and autonomous driving. One solution is to retrieve images with known pose from a database for the localization of query images. However, in environments with drastically varying conditions (e.g. illumination changes, seasons, occlusion, dynamic objects), retrieval-based localization is severely hampered and becomes a challenging problem. In this paper, a novel domain-invariant feature learning method (DIFL) is proposed based on ComboGAN, a multi-domain image translation network architecture. By introducing a feature consistency loss …
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
University of Waterloo（滑铁卢大学）；Georgia Institute of Technology（佐治亚理工学院）；Hong Kong University of Science and Technology（香港科技大学）；Google DeepMind（谷歌DeepMind）
