After a disaster, teams of structural engineers collect vast amounts of images from damaged buildings to obtain lessons and gain knowledge from the event. Images of damaged buildings and components provide valuable evidence to understand the consequences on our structures. However, in many cases, images of damaged buildings are often captured without sufficient spatial context. Also, they may be hard to recognize in cases with severe damage. Incorporating past images showing a pre-disaster condition of such buildings is helpful to accurately evaluate possible circumstances related to a building's failure. One of the best resources to observe the pre-disaster condition of the buildings is Google Street View. A sequence of 360 panorama images which are captured along streets enables all-around views at each location on the street. Once a user knows the GPS information near the building, all external views of the building can be made available. In this study, we develop an automated technique to extract past building images from 360 panorama images serviced by Google Street View. Users only need to provide a geo-tagged image, collected near the target building, and the rest of the process is fully automated. High-quality and undistorted building images are extracted from past panoramas. Since the panoramas are collected from various locations near the building along the street, the user can identify its pre-disaster conditions from the full set of external views.
Hyperlocal disaster damage assessment using bi-temporal street-view imagery and pre-trained vision models
Department of Geography, Texas A&M University（地理系，德克萨斯A&M大学）；Department of Geography, Pennsylvania State University（地理系，宾夕法尼亚州立大学）；Department of Geography and Sustainability, University of Tennessee（地理与可持续发展系，田纳西大学）；Spatial Sciences Institute, University of Southern California（空间科学研究所，南加州大学）
DamageArbiter: A Multimodal Arbitration Framework for Disaster Damage Assessment from Street-View Imagery
organization= Department of Geography, Texas A\&M University , city= College Station , country= USA；organization= Department of Landscape Architecture \& Urban Planning, Texas A\&M University , city= College Station , country= USA；organization= Department of Industrial；Systems Engineering, University of Florida , city= Gainesville , country= USA；organization= Spatial Sciences Institute, University of Southern California , city= Los Angeles , country= USA；organization= Department of Geography；Sustainability, University of Tennessee , city= Knoxville , country= USA
Recov-Vision: Linking Street View Imagery and Vision-Language Models for Post-Disaster Recovery
UrbanResilience.AI Lab（UrbanResilience.AI 实验室）；Zachry Department of Civil and Environmental Engineering（Zachry 土木与环境工程系）；Texas A&M University（德克萨斯A&M大学）；Department of Earth, Marine and Environmental Sciences（地球、海洋与环境科学系）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
The sun glare is one of the major environmental hazards that cause traffic accidents. Every year, many people died and injured in traffic accidents related to sun glare. Providing accurate information about when and where sun glare happens would be helpful to prevent sun glare caused traffic accidents and save lives. In this study, we proposed to use publicly accessible Google Street View (GSV) panorama images to estimate and predict the occurrence of sun glare. GSV images have view sight similar to drivers, which would make GSV images suitable for estimating the visibility of sun glare to dri…
Accurate metrical localization is one of the central challenges in mobile robotics. Many existing methods aim at localizing after building a map with the robot. In this paper, we present a novel approach that instead uses geotagged panoramas from the Google Street View as a source of global positioning. We model the problem of localization as a non-linear least squares estimation in two phases. The first estimates the 3D position of tracked feature points from short monocular camera sequences. The second computes the rigid body transformation between the Street View panoramas and the estimated…
Ground referencing is essential for supervised crop mapping. However, conventional ground truthing involves extensive field surveys and post processing, which is costly in terms of time and labor. In this study, we applied a convolutional neural network (CNN) model to explore the efficacy of automatic ground truthing via Google street view (GSV) images in two distinct farming regions: central Illinois and southern California. We demonstrated the feasibility and reliability of the new ground referencing technique further by performing pixel-based crop mapping with vegetation indices as the mode…
Engineering Vellore Institute of Technology, AP Guntur, India（工程维洛雷理工学院，AP古尔图，印度）；Engineering Vellore Institute of Technology, AP Ongole, India（工程维洛雷理工学院，AP翁戈尔，印度）；Engineering Vellore Institute of Technology, AP Vijayawada, India（工程维洛雷理工学院，AP维杰亚瓦达，印度）
Stanford University（斯坦福大学）；University of Michigan（密歇根大学）；Baylor College of Medicine（贝勒医学院）；Rice University（莱斯大学）
The United States spends more than $1B each year on initiatives such as the American Community Survey (ACS), a labor-intensive door-to-door study that measures statistics relating to race, gender, education, occupation, unemployment, and other demographic factors. Although a comprehensive source of data, the lag between demographic changes and their appearance in the ACS can exceed half a decade. As digital imagery becomes ubiquitous and machine vision techniques improve, automated data analysis may provide a cheaper and faster alternative. Here, we present a method that determines socioeconom…
Cross-View Geolocalization and Disaster Mapping with Street-View and VHR Satellite Imagery: A Case Study of Hurricane IAN
Technical University of Munich（慕尼黑工业大学）；University of the Bundeswehr Munich（慕尼黑联邦国防军大学）；China University of Mining and Technology（中国矿业大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Tongji University（同济大学）
As an important part of urban landscape research, analyzing and studying street-level greenery can increase the understanding of a city's greenery, contributing to better urban living environment planning and design. Planning the best path of urban greenery is a means to effectively maximize the use of urban greenery, which plays a positive role in the physical and mental health of urban residents and the path planning of visitors. In this paper, we used Google Street View (GSV) to obtain street view images of Osaka City. The semantic segmentation model is adopted to segment the street view im…
With the advances of fifth-generation (5G) cellular networks technology, many studies and work have been carried out on how to build 5G networks for smart cities. In the previous research, street lighting poles and smart light poles are capable of being a 5G access point. In order to determine the position of the points, this paper discusses a new way to identify poles based on Mask R-CNN, which extends Fast R-CNNs by making it employ recursive Bayesian filtering and perform proposal propagation and reuse. The dataset contains 3,000 high-resolution images from google map. To make training fast…
Accurate information about the location and orientation of a camera in mobile devices is central to the utilization of location-based services (LBS). Most of such mobile devices rely on GPS data but this data is subject to inaccuracy due to imperfections in the quality of the signal provided by satellites. This shortcoming has spurred the research into improving the accuracy of localization. Since mobile devices have camera, a major thrust of this research has been seeks to acquire the local scene and apply image retrieval techniques by querying a GPS-tagged image database to find the best mat…
National Institute of Technology Surathkal（苏尔塔卡尔国家技术学院）；Insight Centre for Data Analytics, Dublin City University（都柏林城市大学数据分析洞察中心）
Lane detection algorithms have been the key enablers for a fully-assistive and autonomous navigation systems. In this paper, a novel and pragmatic approach for lane detection is proposed using a convolutional neural network (CNN) model based on SegNet encoder-decoder architecture. The encoder block renders low-resolution feature maps of the input and the decoder block provides pixel-wise classification from the feature maps. The proposed model has been trained over 2000 image data-set and tested against their corresponding ground-truth provided in the data-set for evaluation. To enable real-ti…
Maintaining the roadway infrastructure is one of the essential factors in enabling a safe, economic, and sustainable transportation system. Manual roadway damage data collection is laborious and unsafe for humans to perform. This area is poised to benefit from the rapid advance and diffusion of artificial intelligence technologies. Specifically, deep learning advancements enable the detection of road damages automatically from the collected road images. This work proposes to collect and label road damage data using Google Street View and use YOLOv7 (You Only Look Once version 7) together with …
California Institute of Technology（加州理工学院）；ETH Zürich（苏黎世联邦理工学院）
Up-to-date catalogs of the urban tree population are important for municipalities to monitor and improve quality of life in cities. Despite much research on automation of tree mapping, mainly relying on dedicated airborne LiDAR or hyperspectral campaigns, trees are still mostly mapped manually in practice. We present a fully automated tree detection and species recognition pipeline to process thousands of trees within a few hours using publicly available aerial and street view images of Google MapsTM. These data provide rich information (viewpoints, scales) from global tree shapes to bark text…
We introduce the French Street Name Signs (FSNS) Dataset consisting of more than a million images of street name signs cropped from Google Street View images of France. Each image contains several views of the same street name sign. Every image has normalized, title case folded ground-truth text as it would appear on a map. We believe that the FSNS dataset is large and complex enough to train a deep network of significant complexity to solve the street name extraction problem "end-to-end" or to explore the design trade-offs between a single complex engineered network and multiple sub-networks …
ETH Zürich（苏黎世联邦理工学院）；US Department of Agriculture（美国农业部）；University of Edinburgh（爱丁堡大学）；California Institute of Technology（加州理工学院）
We introduce an approach for updating older tree inventories with geographic coordinates using street-level panorama images and a global optimization framework for tree instance matching. Geolocations of trees in inventories until the early 2000s where recorded using street addresses whereas newer inventories use GPS. Our method retrofits older inventories with geographic coordinates to allow connecting them with newer inventories to facilitate long-term studies on tree mortality etc. What makes this problem challenging is the different number of trees per street address, the heterogeneous app…
Multimodal Urban Tree Detection from Satellite and Street-Level Imagery via Annotation-Efficient Deep Learning Strategies
Department of Computer Science, University of California, Davis（加州大学戴维斯分校计算机科学系）；Department of Biological and Agricultural Engineering, University of California, Davis（加州大学戴维斯分校生物与农业工程系）
Decoding the Disaster: Multi-Task Geospatial Reasoning with Vision-Language Models and Crowdsourced Imagery for Disaster Mapping
Shandong University of Science and Technology（山东科技大学）；Technical University of Munich（慕尼黑工业大学）；Wuhan University（武汉大学）；Tsinghua University（清华大学）；Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；University of Iceland（冰岛大学）
University College London（伦敦大学学院）；Google Inc.（谷歌公司）
We present a neural network model - based on CNNs, RNNs and a novel attention mechanism - which achieves 84.2% accuracy on the challenging French Street Name Signs (FSNS) dataset, significantly outperforming the previous state of the art (Smith'16), which achieved 72.46%. Furthermore, our new method is much simpler and more general than the previous approach. To demonstrate the generality of our model, we show that it also performs well on an even more challenging dataset derived from Google Street View, in which the goal is to extract business names from store fronts. Finally, we study the sp…
This paper presents a metric global localization in the urban environment only with a monocular camera and the Google Street View database. We fully leverage the abundant sources from the Street View and benefits from its topo-metric structure to build a coarse-to-fine positioning, namely a topological place recognition process and then a metric pose estimation by local bundle adjustment. Our method is tested on a 3 km urban environment and demonstrates both sub-meter accuracy and robustness to viewpoint changes, illumination and occlusion. To our knowledge, this is the first work that studies…
How to find places that are not indexed by Google Maps? We propose an intuitive method and framework to locate places based on their distinctive spatial features. The method uses satellite and street view images in machine vision approaches to classify locations. If we can classify locations, we just need to repeat for non-overlapping locations in our area of interest. We assess the proposed system in finding Parkour spots in the campus of Arizona State University. The results are very satisfactory, having found more than 25 new Parkour spots, with a rate of true positives above 60%.
Royal Melbourne Institute of Technology（皇家墨尔本理工大学）
On-road bicycle lanes improve safety for cyclists, and encourage participation in cycling for active transport and recreation. With many local authorities responsible for portions of the infrastructure, official maps and datasets of bicycle lanes may be out-of-date and incomplete. Even "crowdsourced" databases may have significant gaps, especially outside popular metropolitan areas. This thesis presents a method to create a map of bicycle lanes in a survey area by taking sample street scene images from each road, and then applying a deep learning model that has been trained to recognise bicycl…
Australian National University（澳大利亚国立大学）；University of Oxford（牛津大学）；University of Technology Sydney（悉尼科技大学）
This paper presents a new approach for synthesizing a novel street-view panorama given an overhead satellite image. Taking a small satellite image patch as input, our method generates a Google's omnidirectional street-view type panorama, as if it is captured from the same geographical location as the center of the satellite patch. Existing works tackle this task as an image generation problem which adopts generative adversarial networks to implicitly learn the cross-view transformations, while ignoring the domain relevance. In this paper, we propose to explicitly establish the geometric corres…
School of Mechanical Engineering, Purdue University（普渡大学机械工程学院）；Department of Civil and Environmental Engineering, University of Waterloo（滑铁卢大学土木与环境工程系）；Lyles School of Civil Engineering, Purdue University（普渡大学莱尔土木工程学院）
After a disaster, teams of structural engineers collect vast amounts of images from damaged buildings to obtain new knowledge and extract lessons from the event. However, in many cases, the images collected are captured without sufficient spatial context. When damage is severe, it may be quite difficult to even recognize the building. Accessing images of the pre-disaster condition of those buildings is required to accurately identify the cause of the failure or the actual loss in the building. Here, to address this issue, we develop a method to automatically extract pre-event building images f…
Technical University of Munich (TUM)（慕尼黑工业大学）；German Aerospace Center (DLR)（德国航空航天中心）
Land-use classification based on spaceborne or aerial remote sensing images has been extensively studied over the past decades. Such classification is usually a patch-wise or pixel-wise labeling over the whole image. But for many applications, such as urban population density mapping or urban utility planning, a classification map based on individual buildings is much more informative. However, such semantic classification still poses some fundamental challenges, for example, how to retrieve fine boundaries of individual buildings. In this paper, we proposed a general framework for classifying…
Today's autonomous vehicles rely extensively on high-definition 3D maps to navigate the environment. While this approach works well when these maps are completely up-to-date, safe autonomous vehicles must be able to corroborate the map's information via a real time sensor-based system. Our goal in this work is to develop a model for road layout inference given imagery from on-board cameras, without any reliance on high-definition maps. However, no sufficient dataset for training such a model exists. Here, we leverage the availability of standard navigation maps and corresponding street view im…
Beyond Geo-localization: Fine-grained Orientation of Street-view Images by Cross-view Matching with Satellite Imagery with Supplementary Materials
National University of Singapore（新加坡国立大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Grab Chronos S.R.L.（Grab Chronos有限责任公司）；Grabtaxi Holdings Pte. Ltd.（Grab出租车控股私人有限公司）
Chinese University of Hong Kong（香港中文大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
The ability to recognize the position and order of the floor-level lines that divide adjacent building floors can benefit many applications, for example, urban augmented reality (AR). This work tackles the problem of locating floor-level lines in street-view images, using a supervised deep learning approach. Unfortunately, very little data is available for training such a network $-$ current street-view datasets contain either semantic annotations that lack geometric attributes, or rectified facades without perspective priors. To address this issue, we first compile a new dataset and develop a…
Cross-View Geo-Localization with Street-View and VHR Satellite Imagery in Decentrality Settings
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；Ant Group（蚂蚁集团）；Technology Innovation Center for Collaborative Applications of Natural Resources Data in GBA, Ministry of Natural Resources（自然资源部粤港澳自然资源数据协同应用技术创新中心）
Navigating and understanding the real world remains a key challenge in machine learning and inspires a great variety of research in areas such as language grounding, planning, navigation and computer vision. We propose an instruction-following task that requires all of the above, and which combines the practicality of simulated environments with the challenges of ambiguous, noisy real world data. StreetNav is built on top of Google Street View and provides visually accurate environments representing real places. Agents are given driving instructions which they must learn to interpret in order …
DisasterTD: Disaster Toponym Disambiguation Using Multimodal LLMs and Cross-View Geolocalization
College of Geodesy and Geomatics, Shandong University of Science and Technology（山东科技大学测绘与地理信息学院）；School of Environmental Science and Spatial Informatics, China University of Mining and Technology（中国矿业大学环境与测绘学院）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Automation, Southeast University（东南大学自动化学院）；Department of Geography, National University of Singapore（新加坡国立大学地理系）
SkyLink: Unifying Street-Satellite Geo-Localization via UAV-Mediated 3D Scene Alignment
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，教育部，厦门大学）；School of Electronic and Information Engineering, Foshan University（电子与信息工程学院，佛山大学）
We introduce Chinese Text in the Wild, a very large dataset of Chinese text in street view images. While optical character recognition (OCR) in document images is well studied and many commercial tools are available, detection and recognition of text in natural images is still a challenging problem, especially for more complicated character sets such as Chinese text. Lack of training data has always been a problem, especially for deep learning methods which require massive training data. In this paper we provide details of a newly created dataset of Chinese text with about 1 million Chinese ch…
Stanford University（斯坦福大学）；University of Michigan（密歇根大学）
Targeted socioeconomic policies require an accurate understanding of a country's demographic makeup. To that end, the United States spends more than 1 billion dollars a year gathering census data such as race, gender, education, occupation and unemployment rates. Compared to the traditional method of collecting surveys across many years which is costly and labor intensive, data-driven, machine learning driven approaches are cheaper and faster--with the potential ability to detect trends in close to real time. In this work, we leverage the ubiquity of Google Street View images and develop a com…
LLMs as World Models: Data-Driven and Human-Centered Pre-Event Simulation for Disaster Impact Assessment
University of South Florida（佛罗里达州立大学）；Arizona State University（亚利桑那州立大学）；Massachusetts Institute of Technology（麻省理工学院）；New York University（纽约大学）；University of Alabama（阿拉巴马大学）；Texas Tech University（德克萨斯科技大学）
Recent advances in deep learning have made it possible to quantify urban metrics at fine resolution, and over large extents using street-level images. Here, we focus on measuring urban tree cover using Google Street View (GSV) images. First, we provide a small-scale labelled validation dataset and propose standard metrics to compare the performance of automated estimations of street tree cover using GSV. We apply state-of-the-art deep learning models, and compare their performance to a previously established benchmark of an unsupervised method. Our training procedure for deep learning models i…
We present a novel approach to geolocalising panoramic images on a 2-D cartographic map based on learning a low dimensional embedded space, which allows a comparison between an image captured at a location and local neighbourhoods of the map. The representation is not sufficiently discriminatory to allow localisation from a single image, but when concatenated along a route, localisation converges quickly, with over 90% accuracy being achieved for routes of around 200m in length when using Google Street View and Open Street Map data. The method generalises a previous fixed semantic feature base…
BuildingView: Constructing Urban Building Exteriors Databases with Street View Imagery and Multimodal Large Language Mode
Thrust of Urban Governance and Design, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）城市治理与设计中心）；Spatial Sciences Institute, University of Southern California（南加州大学空间科学学院）
