Geo-localizing static objects from street images is challenging but also very important for road asset mapping and autonomous driving. In this paper we present a two-stage framework that detects and geolocalizes traffic signs from low frame rate street videos. Our proposed system uses a modified version of RetinaNet (GPS-RetinaNet), which predicts a positional offset for each sign relative to the camera, in addition to performing the standard classification and bounding box regression. Candidate sign detections from GPS-RetinaNet are condensed into geolocalized signs by our custom tracker, which consists of a learned metric network and a variant of the Hungarian Algorithm. Our metric network estimates the similarity between pairs of detections, then the Hungarian Algorithm matches detections across images using the similarity scores provided by the metric network. Our models were trained using an updated version of the ARTS dataset, which contains 25,544 images and 47.589 sign annotations ~\cite{arts}. The proposed dataset covers a diverse set of environments gathered from a broad selection of roads. Each annotaiton contains a sign class label, its geospatial location, an assembly label, a side of road indicator, and unique identifiers that aid in the evaluation. This dataset will support future progress in the field, and the proposed system demonstrates how to take advantage of some of the unique characteristics of a realistic geolocalization dataset.
Accurate pose estimation is a fundamental ability that all mobile robots must posses in order to traverse robustly in a given environment. Much like a human, this ability is dependent on the robot's understanding of a given scene. For Autonomous Vehicles (AV's), detailed 3D maps created beforehand are widely used to augment the perceptive abilities and estimate pose based on current sensor measurements. This approach however is less suited for rural communities that are sparsely connected and cover large areas. To deal with the challenge of localizing a vehicle in a rural setting, this paper p…
Institute for Transport Studies (ITS), University of Leeds（利兹大学交通研究所）；Lincoln Centre for Autonomous Systems, University of Lincoln（林肯大学林肯自主系统中心）；Institute of Engineering Sciences, De Montfort University（德蒙福特大学工程科学研究所）；School of Mechanical Engineering, Nat. Tech. University of Athens（雅典国家技术大学机械工程学院）；Robert Bosch GmbH（罗伯特·博世有限公司）；Ibex Automation Ltd（伊贝克斯自动化有限公司）
Autonomous vehicles (AVs) must share space with pedestrians, both in carriageway cases such as cars at pedestrian crossings and off-carriageway cases such as delivery vehicles navigating through crowds on pedestrianized high-streets. Unlike static obstacles, pedestrians are active agents with complex, interactive motions. Planning AV actions in the presence of pedestrians thus requires modelling of their probable future behaviour as well as detecting and tracking them. This narrative review article is Part I of a pair, together surveying the current technology stack involved in this process, o…
We present an approach towards robust lane tracking for assisted and autonomous driving, particularly under poor visibility. Autonomous detection of lane markers improves road safety, and purely visual tracking is desirable for widespread vehicle compatibility and reducing sensor intrusion, cost, and energy consumption. However, visual approaches are often ineffective because of a number of factors, including but not limited to occlusion, poor weather conditions, and paint wear-off. Our method, named SafeDrive, attempts to improve visual lane detection approaches in drastically degraded visual…
Mapillary Research（Mapillary研究中心）；Graz University of Technology（格拉茨理工大学）；Computer Vision Center, UAB（巴塞罗那自治大学计算机视觉中心）
In this work we contribute a novel pipeline to automatically generate training data, and to improve over state-of-the-art multi-object tracking and segmentation (MOTS) methods. Our proposed track mining algorithm turns raw street-level videos into high-fidelity MOTS training data, is scalable and overcomes the need of expensive and time-consuming manual annotation approaches. We leverage state-of-the-art instance segmentation results in combination with optical flow predictions, also trained on automatically harvested training data. Our second major contribution is MOTSNet - a deep learning, t…
University of Science and Technology of China（中国科学技术大学）；The Chinese Academy of Sciences（中国科学院）
Multiple-object tracking and behavior analysis have been the essential parts of surveillance video analysis for public security and urban management. With billions of surveillance video captured all over the world, multiple-object tracking and behavior analysis by manual labor are cumbersome and cost expensive. Due to the rapid development of deep learning algorithms in recent years, automatic object tracking and behavior analysis put forward an urgent demand on a large scale well-annotated surveillance video dataset that can reflect the diverse, congested, and complicated scenarios in real ap…
The Alan Turing Institute（艾伦·图灵研究所）；University College London（伦敦大学学院）
Recent researches have shown the increasing use of machine learn-ing methods in geography and urban analytics, primarily to extract features and patterns from spatial and temporal data using a supervised approach. Researches integrating geographical processes in machine learning models and the use of unsupervised approacheson geographical data for knowledge discovery had been sparse. This research contributes to the ladder, where we show how latent variables learned from unsupervised learning methods on urbanimages can be used for geographic knowledge discovery. In particular, we propose a sim…
GeoReasoner: Geo-localization with Reasoning in Street Views using a Large Vision-Language Model
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tongji University（同济大学）；Independent Researcher（独立研究者）；Information Engineering University（信息工程大学）
Algolux；Mercedes-Benz AG（梅赛德斯-奔驰股份公司）；Princeton University（普林斯顿大学）；Ulm University（乌尔姆大学）；University of Kassel（卡塞尔大学）
Conventional sensor systems record information about directly visible objects, whereas occluded scene components are considered lost in the measurement process. Non-line-of-sight (NLOS) methods try to recover such hidden objects from their indirect reflections - faint signal components, traditionally treated as measurement noise. Existing NLOS approaches struggle to record these low-signal components outside the lab, and do not scale to large-scale outdoor scenes and high-speed motion, typical in automotive scenarios. In particular, optical NLOS capture is fundamentally limited by the quartic …
ETH Zürich（苏黎世联邦理工学院）；US Department of Agriculture（美国农业部）；University of Edinburgh（爱丁堡大学）；California Institute of Technology（加州理工学院）
We introduce an approach for updating older tree inventories with geographic coordinates using street-level panorama images and a global optimization framework for tree instance matching. Geolocations of trees in inventories until the early 2000s where recorded using street addresses whereas newer inventories use GPS. Our method retrofits older inventories with geographic coordinates to allow connecting them with newer inventories to facilitate long-term studies on tree mortality etc. What makes this problem challenging is the different number of trees per street address, the heterogeneous app…
The goal of cross-view image based geo-localization is to determine the location of a given street view image by matching it against a collection of geo-tagged satellite images. This task is notoriously challenging due to the drastic viewpoint and appearance differences between the two domains. We show that we can address this discrepancy explicitly by learning to synthesize realistic street views from satellite inputs. Following this observation, we propose a novel multi-task architecture in which image synthesis and retrieval are considered jointly. The rationale behind this is that we can b…
Australian National University（澳大利亚国立大学）；University of Oxford（牛津大学）；University of Technology Sydney（悉尼科技大学）
This paper presents a new approach for synthesizing a novel street-view panorama given an overhead satellite image. Taking a small satellite image patch as input, our method generates a Google's omnidirectional street-view type panorama, as if it is captured from the same geographical location as the center of the satellite patch. Existing works tackle this task as an image generation problem which adopts generative adversarial networks to implicitly learn the cross-view transformations, while ignoring the domain relevance. In this paper, we propose to explicitly establish the geometric corres…
Northwestern Polytechnical University（西北工业大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）
This paper is concerned with the problem of how to better exploit 3D geometric information for dense semantic image labeling. Existing methods often treat the available 3D geometry information (e.g., 3D depth-map) simply as an additional image channel besides the R-G-B color channels, and apply the same technique for RGB image labeling. In this paper, we demonstrate that directly performing 3D convolution in the framework of a residual connected 3D voxel top-down modulation network can lead to superior results. Specifically, we propose a 3D semantic labeling method to label outdoor street scen…
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
Street-to-aerial image geo-localization, which matches a query street-view image to the GPS-tagged aerial images in a reference set, has attracted increasing attention recently. In this paper, we revisit this problem and point out the ignored issue about image alignment information. We show that the performance of a simple Siamese network is highly dependent on the alignment setting and the comparison of previous works can be unfair if they have different assumptions. Instead of focusing on the feature extraction under the alignment assumption, we show that improvements in metric learning tech…
Beyond Geo-localization: Fine-grained Orientation of Street-view Images by Cross-view Matching with Satellite Imagery with Supplementary Materials
National University of Singapore（新加坡国立大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Grab Chronos S.R.L.（Grab Chronos有限责任公司）；Grabtaxi Holdings Pte. Ltd.（Grab出租车控股私人有限公司）
ADAPT Centre（ADAPT中心）；School of Computer Science and Statistics, Trinity College Dublin（都柏林圣三一大学计算机科学与统计学学院）
Many applications such as autonomous navigation, urban planning and asset monitoring, rely on the availability of accurate information about objects and their geolocations. In this paper we propose to automatically detect and compute the GPS coordinates of recurring stationary objects of interest using street view imagery. Our processing pipeline relies on two fully convolutional neural networks: the first segments objects in the images while the second estimates their distance from the camera. To geolocate all the detected objects coherently we propose a novel custom Markov Random Field model…
Spatially-Weighted CLIP for Street-View Geo-localization
School of Geospatial Engineering and Science, Sun Yat-Sen University（中山大学地理科学与规划学院）；School of Geographical and Earth Sciences, University of Glasgow（格拉斯哥大学地理与地球科学学院）；School of System Science and Engineering, Sun Yat-Sen University（中山大学系统科学与工程学院）
DSTO（澳大利亚国防科学与技术组织）；Chalmers University of Technology（查尔姆斯理工大学）
The paper evaluates the error performance of three random finite set based multi-object trackers in the context of pedestrian video tracking. The evaluation is carried out using a publicly available video dataset of 4500 frames (town centre street) for which the ground truth is available. The input to all pedestrian tracking algorithms is an identical set of head and body detections, obtained using the Histogram of Oriented Gradients (HOG) detector. The tracking error is measured using the recently proposed OSPA metric for tracks, adopted as the only known mathematically rigorous metric for me…
SkyLink: Unifying Street-Satellite Geo-Localization via UAV-Mediated 3D Scene Alignment
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，教育部，厦门大学）；School of Electronic and Information Engineering, Foshan University（电子与信息工程学院，佛山大学）
ETH Zurich（苏黎世联邦理工学院）；The University of Tokyo（东京大学）；Zhejiang University（浙江大学）；The Ohio State University（俄亥俄州立大学）；Microsoft（微软）
We present a novel method for synthesizing both temporally and geometrically consistent street-view panoramic video from a single satellite image and camera trajectory. Existing cross-view synthesis approaches focus on images, while video synthesis in such a case has not yet received enough attention. For geometrical and temporal consistency, our approach explicitly creates a 3D point cloud representation of the scene and maintains dense 3D-2D correspondences across frames that reflect the geometric scene configuration inferred from the satellite view. As for synthesis in the 3D space, we impl…
From Satellite to Street: A Hybrid Framework Integrating Stable Diffusion and PanoGAN for Consistent Cross-View Synthesis
Department of Information and Computer Science, King Fahd University of Petroleum and Minerals（信息与计算机科学系，国王法赫德石油与矿物大学）；NCMI, CSIRO（CSIRO国家科学机构）；Department of Computer Science, National University of Computer and Emerging Sciences (FAST-NUCES)（计算机科学系，国家计算机与新兴科学大学（FAST-NUCES））；Faculty of Engineering and IT, University of Technology Sydney（工程与信息技术学院，技术大学悉尼）；Faculty of Science, University of Ontario Institute of Technology（科学学院， Ontario Institute of Technology大学）
针对卫星到街景跨视角合成中的视角与外观差异，本文提出融合 Stable Diffusion、条件 GAN 与双分支多阶段训练的混合框架，在 CVUSA 上提升几何一致性与视觉真实感。
Massachusetts Institute of Technology（麻省理工学院）；Draper（德雷珀实验室）
Cross-view image geolocalization provides an estimate of an agent's global position by matching a local ground image to an overhead satellite image without the need for GPS. It is challenging to reliably match a ground image to the correct satellite image since the images have significant viewpoint differences. Existing works have demonstrated localization in constrained scenarios over small areas but have not demonstrated wider-scale localization. Our approach, called Wide-Area Geolocalization (WAG), combines a neural network with a particle filter to achieve global position estimates for age…
Accurate metrical localization is one of the central challenges in mobile robotics. Many existing methods aim at localizing after building a map with the robot. In this paper, we present a novel approach that instead uses geotagged panoramas from the Google Street View as a source of global positioning. We model the problem of localization as a non-linear least squares estimation in two phases. The first estimates the 3D position of tracked feature points from short monocular camera sequences. The second computes the rigid body transformation between the Street View panoramas and the estimated…
Fine-Grained Building Function Recognition from Street-View Images via Geometry-Aware Semi-Supervised Learning
School of Geospatial Engineering and Science, Sun Yat-sen University（中山大学地理空间工程与科学学院）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
University of Washington（华盛顿大学）；Google Research（谷歌研究院）；Stanford University（斯坦福大学）
We present a framework for automatically reconfiguring images of street scenes by populating, depopulating, or repopulating them with objects such as pedestrians or vehicles. Applications of this method include anonymizing images to enhance privacy, generating data augmentations for perception tasks like autonomous driving, and composing scenes to achieve a certain ambiance, such as empty streets in the early morning. At a technical level, our work has three primary contributions: (1) a method for clearing images of objects, (2) a method for estimating sun direction from a single image, and (3…
Cross-View Geo-Localization with Street-View and VHR Satellite Imagery in Decentrality Settings
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；Ant Group（蚂蚁集团）；Technology Innovation Center for Collaborative Applications of Natural Resources Data in GBA, Ministry of Natural Resources（自然资源部粤港澳自然资源数据协同应用技术创新中心）
Street-Level Geolocalization Using Multimodal Large Language Models and Retrieval-Augmented Generation
Vocational School of Social Sciences, Marmara University（马尔马拉大学社会科学职业学校）；Geospatial Data Science Group, School of Geographical & Earth Sciences, University of Glasgow（格拉斯哥大学地理与地球科学学院空间数据科学小组）
Department of Information and Computer Science（信息与计算机科学系）；King Fahd University of Petroleum and Minerals（国王法赫德石油与矿物大学）；SDAIA-KFUPM Joint Research Center of Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）；Ontario Tech University（安大略技术大学）；University of Canberra（堪培拉大学）；University of Regina（里贾纳大学）
More than half of the world's roads lack adequate street addressing systems. Lack of addresses is even more visible in daily lives of people in developing countries. We would like to object to the assumption that having an address is a luxury, by proposing a generative address design that maps the world in accordance with streets. The addressing scheme is designed considering several traditional street addressing methodologies employed in the urban development scenarios around the world. Our algorithm applies deep learning to extract roads from satellite images, converts the road pixel confide…
We present a novel approach to geolocalising panoramic images on a 2-D cartographic map based on learning a low dimensional embedded space, which allows a comparison between an image captured at a location and local neighbourhoods of the map. The representation is not sufficiently discriminatory to allow localisation from a single image, but when concatenated along a route, localisation converges quickly, with over 90% accuracy being achieved for routes of around 200m in length when using Google Street View and Open Street Map data. The method generalises a previous fixed semantic feature base…
TrajGANR: Trajectory-Centric Urban Multimodal Learning via Geospatially Aligned Neural Representations
Google Research, Mountain View, CA（谷歌研究，山景城，加利福尼亚州）；Google LLC, Mountain View, CA（谷歌公司，山景城，加利福尼亚州）；Dept. of Computer Science, University of Southern California, Los Angeles, CA（计算机科学系，南加州大学，洛杉矶，加利福尼亚州）；SEAI Lab, Dept. of Geography and the Environment, The University of Texas at Austin, Austin, TX（SEAI实验室，地理与环境系，德克萨斯大学奥斯汀分校，奥斯汀，德克萨斯州）
Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Computer Vision Center（计算机视觉中心）；Daimler AG, R&D（戴姆勒股份公司研发部）；ETH Zürich（苏黎世联邦理工学院）
In this work we present a novel compact scene representation based on Stixels that infers geometric and semantic information. Our approach overcomes the previous rather restrictive geometric assumptions for Stixels by introducing a novel depth model to account for non-flat roads and slanted objects. Both semantic and depth cues are used jointly to infer the scene representation in a sound global energy minimization formulation. Furthermore, a novel approximation scheme is introduced that uses an extremely efficient over-segmentation. In doing so, the computational complexity of the Stixel infe…
Tracking in urban street scenes plays a central role in autonomous systems such as self-driving cars. Most of the current vision-based tracking methods perform tracking in the image domain. Other approaches, eg based on LIDAR and radar, track purely in 3D. While some vision-based tracking methods invoke 3D information in parts of their pipeline, and some 3D-based methods utilize image-based information in components of their approach, we propose to use image- and world-space information jointly throughout our method. We present our tracking pipeline as a 3D extension of image-based tracking. F…
From Street to Orbit: Training-Free Cross-View Retrieval via Location Semantics and LLM Guidance
Ulsan National Institute of Science and Technology (UNIST)（蔚山科学技术院）；Electronics and Telecommunications Research Institute (ETRI)（电子通信研究院）；Kyungpook National University（庆北大学）
In this paper, based on the assumption that the object boundaries (e.g., buildings) from the over-view data should coincide with footprints of façade 3D points generated from street-view photogrammetric images, we aim to address this problem by proposing a fully automated geo-registration method for cross-view data, which utilizes semantically segmented object boundaries as view-invariant features under a global optimization framework through graph-matching: taking the over-view point clouds generated from stereo/multi-stereo satellite images and the street-view point clouds generated from mon…
Extracting information related to weather and visual conditions at a given time and space is indispensable for scene awareness, which strongly impacts our behaviours, from simply walking in a city to riding a bike, driving a car, or autonomous drive-assistance. Despite the significance of this subject, it is still not been fully addressed by the machine intelligence relying on deep learning and computer vision to detect the multi-labels of weather and visual conditions with a unified method that can be easily used for practice. What has been achieved to-date is rather sectorial models that add…
