Object detection and classification of traffic signs in street-view imagery is an essential element for asset management, map making and autonomous driving. However, some traffic signs occur rarely and consequently, they are difficult to recognize automatically. To improve the detection and classification rates, we propose to generate images of traffic signs, which are then used to train a detector/classifier. In this research, we present an end-to-end framework that generates a realistic image of a traffic sign from a given image of a traffic sign and a pictogram of the target class. We propose a residual attention mechanism with dense concatenation called Dense Residual Attention, that preserves the background information while transferring the object information. We also propose to utilize multi-scale discriminators, so that the smaller scales of the output guide the higher resolution output. We have performed detection and classification tests across a large number of traffic sign classes, by training the detector using the combination of real and generated data. The newly trained model reduces the number of false positives by 1.2 - 1.5% at 99% recall in the detection tests and an absolute improvement of 4.65% (top-1 accuracy) in the classification tests.
Massachusetts Institute of Technology（麻省理工学院）；University of California, Berkeley（加州大学伯克利分校）
The classification of streets on road networks has been focused on the vehicular transportational features of streets such as arterials, major roads, minor roads and so forth based on their transportational use. City authorities on the other hand have been shifting to more urban inclusive planning of streets, encompassing the side use of a street combined with the transportational features of a street. In such classification schemes, streets are labeled for example as commercial throughway, residential neighborhood, park etc. This modern approach to urban planning has been adopted by major cit…
The ability to navigate from visual observations in unfamiliar environments is a core component of intelligent agents and an ongoing challenge for Deep Reinforcement Learning (RL). Street View can be a sensible testbed for such RL agents, because it provides real-world photographic imagery at ground level, with diverse street appearances; it has been made into an interactive environment called StreetLearn and used for research on navigation. However, goal-driven street navigation agents have not so far been able to transfer to unseen areas without extensive retraining, and relying on simulatio…
Today's autonomous vehicles rely extensively on high-definition 3D maps to navigate the environment. While this approach works well when these maps are completely up-to-date, safe autonomous vehicles must be able to corroborate the map's information via a real time sensor-based system. Our goal in this work is to develop a model for road layout inference given imagery from on-board cameras, without any reliance on high-definition maps. However, no sufficient dataset for training such a model exists. Here, we leverage the availability of standard navigation maps and corresponding street view im…
SA-Occ: Satellite-Assisted 3D Occupancy Prediction in Real World
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
Designing streetscapes from street-view imagery using diffusion models
Department of Urban and Regional Planning, University of Florida（城市与区域规划系，佛罗里达大学）；Singapore-MIT Alliance for Research and Technology Centre (SMART)（新加坡-麻省理工联合研究中心（SMART））；Department of Landscape Architecture and Urban Planning, Texas A&M University（景观建筑与城市规划系，德克萨斯大学安德森分校）；Department of Agronomy, University of Florida（农业系，佛罗里达大学）
Major decisions from governments and other large organizations rely on measurements of the populace's well-being, but making such measurements at a broad scale is expensive and thus infrequent in much of the developing world. We propose an inexpensive, scalable, and interpretable approach to predict key livelihood indicators from public crowd-sourced street-level imagery. Such imagery can be cheaply collected and more frequently updated compared to traditional surveying methods, while containing plausibly relevant information for a range of livelihood indicators. We propose two approaches to l…
TU Delft（代尔夫特理工大学）；TU Eindhoven（埃因霍温理工大学）；Cyclomedia B.V（Cyclomedia公司）
The current paradigm in privacy protection in street-view images is to detect and blur sensitive information. In this paper, we propose a framework that is an alternative to blurring, which automatically removes and inpaints moving objects (e.g. pedestrians, vehicles) in street-view imagery. We propose a novel moving object segmentation algorithm exploiting consistencies in depth across multiple street-view images that are later combined with the results of a segmentation network. The detected moving objects are removed and inpainted with information from other views, to obtain a realistic out…
University College London（伦敦大学学院）；Google Inc.（谷歌公司）
We present a neural network model - based on CNNs, RNNs and a novel attention mechanism - which achieves 84.2% accuracy on the challenging French Street Name Signs (FSNS) dataset, significantly outperforming the previous state of the art (Smith'16), which achieved 72.46%. Furthermore, our new method is much simpler and more general than the previous approach. To demonstrate the generality of our model, we show that it also performs well on an even more challenging dataset derived from Google Street View, in which the goal is to extract business names from store fronts. Finally, we study the sp…
ADAPT Centre（ADAPT中心）；School of Computer Science and Statistics, Trinity College Dublin（都柏林圣三一大学计算机科学与统计学学院）
Many applications such as autonomous navigation, urban planning and asset monitoring, rely on the availability of accurate information about objects and their geolocations. In this paper we propose to automatically detect and compute the GPS coordinates of recurring stationary objects of interest using street view imagery. Our processing pipeline relies on two fully convolutional neural networks: the first segments objects in the images while the second estimates their distance from the camera. To geolocate all the detected objects coherently we propose a novel custom Markov Random Field model…
Efficient Cross-Country Data Acquisition Strategy for ADAS via Street-View Imagery
CARIAD SE；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Esslingen University of Applied Sciences（埃森哲应用科学大学）；FZI Research Center for Information Technology（弗劳恩霍夫信息技术研究中心）
Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural network that operates directly on the image pixels. We employ the DistBelief implementation of deep neural networks…
In this paper we aim to determine the location and orientation of a ground-level query image by matching to a reference database of overhead (e.g. satellite) images. For this task we collect a new dataset with one million pairs of street view and overhead images sampled from eleven U.S. cities. We explore several deep CNN architectures for cross-domain matching -- Classification, Hybrid, Siamese, and Triplet networks. Classification and Hybrid architectures are accurate but slow since they allow only partial feature precomputation. We propose a new loss function which significantly improves th…
Space2Ground 2.0: A Multi-Source Dataset and Framework for Agricultural Monitoring through Fusion of Street-Level and Satellite Imagery
Operational Unit BEYOND Centre, IAASARS, National Observatory of Athens（雅典国家天文台IAASARS研究所BEYOND中心业务单元）；DHI Water & Environment（DHI水环境公司）；Artificial Intelligence Group, Wageningen University & Research（瓦赫宁根大学及研究中心人工智能组）
Cross-Modal Urban Sensing: Evaluating Sound-Vision Alignment Across Street-Level and Aerial Imagery
Department of Geography, University of South Carolina（南卡罗来纳大学地理系）；Department of Environmental Sciences, Emory University（埃默里大学环境科学系）；University of Canterbury（坎特伯雷大学）；School of Resource and Environmental Sciences, Wuhan University（武汉大学资源与环境科学学院）
Australian National University（澳大利亚国立大学）；University of Oxford（牛津大学）；University of Technology Sydney（悉尼科技大学）
This paper presents a new approach for synthesizing a novel street-view panorama given an overhead satellite image. Taking a small satellite image patch as input, our method generates a Google's omnidirectional street-view type panorama, as if it is captured from the same geographical location as the center of the satellite patch. Existing works tackle this task as an image generation problem which adopts generative adversarial networks to implicitly learn the cross-view transformations, while ignoring the domain relevance. In this paper, we propose to explicitly establish the geometric corres…
Studies evaluating bikeability usually compute spatial indicators shaping cycling conditions and conflate them in a quantitative index. Much research involves site visits or conventional geospatial approaches, and few studies have leveraged street view imagery (SVI) for conducting virtual audits. These have assessed a limited range of aspects, and not all have been automated using computer vision (CV). Furthermore, studies have not yet zeroed in on gauging the usability of these technologies thoroughly. We investigate, with experiments at a fine spatial scale and across multiple geographies (S…
Search with local intent is becoming increasingly useful due to the popularity of the mobile device. The creation and maintenance of accurate listings of local businesses worldwide is time consuming and expensive. In this paper, we propose an approach to automatically discover businesses that are visible on street level imagery. Precise business store front detection enables accurate geo-location of businesses, and further provides input for business categorization, listing generation, etc. The large variety of business categories in different countries makes this a very challenging problem. M…
Accurately maintaining digital street maps is labor-intensive. To address this challenge, much work has studied automatically processing geospatial data sources such as GPS trajectories and satellite images to reduce the cost of maintaining digital maps. An end-to-end map update system would first process geospatial data sources to extract insights, and second leverage those insights to update and improve the map. However, prior work largely focuses on the first step of this pipeline: these map extraction methods infer road networks from scratch given geospatial data sources (in effect creatin…
Beyond Geo-localization: Fine-grained Orientation of Street-view Images by Cross-view Matching with Satellite Imagery with Supplementary Materials
National University of Singapore（新加坡国立大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Grab Chronos S.R.L.（Grab Chronos有限责任公司）；Grabtaxi Holdings Pte. Ltd.（Grab出租车控股私人有限公司）
DM-KG: A Novel Method for Boosting Spatial Cognition of Vision-Language Models in Street View Imagery
Institute of Surveying and Mapping, Information Engineering University（信息工程大学测绘学院）；Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences（中国科学院地理科学与资源研究所）
Recov-Vision: Linking Street View Imagery and Vision-Language Models for Post-Disaster Recovery
UrbanResilience.AI Lab（UrbanResilience.AI 实验室）；Zachry Department of Civil and Environmental Engineering（Zachry 土木与环境工程系）；Texas A&M University（德克萨斯A&M大学）；Department of Earth, Marine and Environmental Sciences（地球、海洋与环境科学系）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Hyperlocal disaster damage assessment using bi-temporal street-view imagery and pre-trained vision models
Department of Geography, Texas A&M University（地理系，德克萨斯A&M大学）；Department of Geography, Pennsylvania State University（地理系，宾夕法尼亚州立大学）；Department of Geography and Sustainability, University of Tennessee（地理与可持续发展系，田纳西大学）；Spatial Sciences Institute, University of Southern California（空间科学研究所，南加州大学）
Multimodal Urban Tree Detection from Satellite and Street-Level Imagery via Annotation-Efficient Deep Learning Strategies
Department of Computer Science, University of California, Davis（加州大学戴维斯分校计算机科学系）；Department of Biological and Agricultural Engineering, University of California, Davis（加州大学戴维斯分校生物与农业工程系）
DamageArbiter: A Multimodal Arbitration Framework for Disaster Damage Assessment from Street-View Imagery
organization= Department of Geography, Texas A\&M University , city= College Station , country= USA；organization= Department of Landscape Architecture \& Urban Planning, Texas A\&M University , city= College Station , country= USA；organization= Department of Industrial；Systems Engineering, University of Florida , city= Gainesville , country= USA；organization= Spatial Sciences Institute, University of Southern California , city= Los Angeles , country= USA；organization= Department of Geography；Sustainability, University of Tennessee , city= Knoxville , country= USA
OpenFACADES: An Open Framework for Architectural Caption and Attribute Data Enrichment via Street View Imagery
Department of Architecture, National University of Singapore（建筑系，新加坡国立大学）；Department of Electrical and Computer Engineering, National University of Singapore（电气与计算机工程系，新加坡国立大学）；School of Artificial Intelligence, Shenzhen Technology University（人工智能学院，深圳科技大学）；Department of Real Estate, National University of Singapore（房地产系，新加坡国立大学）
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Qatar Computer Research Institute（卡塔尔计算机研究所）
Street maps are a crucial data source that help to inform a wide range of decisions, from navigating a city to disaster relief and urban planning. However, in many parts of the world, street maps are incomplete or lag behind new construction. Editing maps today involves a tedious process of manually tracing and annotating roads, buildings, and other map features. Over the past decade, many automatic map inference systems have been proposed to automatically extract street map data from satellite imagery, aerial imagery, and GPS trajectory datasets. However, automatic map inference has failed to…
SVII-3D: Advancing Roadside Infrastructure Inventory with Decimeter-level 3D Localization and Comprehension from Sparse Street Imagery
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan 430079, China（信息工程测绘遥感国家重点实验室（LIESMARS），武汉大学）；Research Institute Ltd, Chengdu 610000, China（四川省公路规划设计研究有限公司）
Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagery
Guanghua School of Management, Peking University（北京大学光华管理学院）；The Bartlett School of Sustainable Construction, University College London（伦敦大学学院巴特lett可持续建筑学院）；School of Architecture, Tsinghua University（清华大学建筑学院）；School of Computing, The University of Georgia（佐治亚大学计算学院）；Department of Geography, University College London（伦敦大学学院地理系）；Department of Geography and the Environment, University of Texas at Austin（德克萨斯大学奥斯汀分校地理与环境系）
Stanford University（斯坦福大学）；IBM Research Europe（IBM欧洲研究院）；IBM Europe（IBM欧洲）
Current methods to determine the energy efficiency of buildings require on-site visits of certified energy auditors which makes the process slow, costly, and geographically incomplete. To accelerate the identification of promising retrofit targets on a large scale, we propose to estimate building energy efficiency from widely available and remotely sensed data sources only, namely street view, aerial view, footprint, and satellite-borne land surface temperature (LST) data. After collecting data for almost 40,000 buildings in the United Kingdom, we combine these data sources by training multipl…
How does spatial structure affect psychological restoration? A method based on Graph Neural Networks and Street View Imagery
Hong Kong University of Science and Technology（香港科技大学）；Jiangnan University（江南大学）；The Chinese University of Hong Kong（香港中文大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；Peking University（北京大学）
UPDExplainer: an Interpretable Transformer-based Framework for Urban Physical Disorder Detection Using Street View Imagery
West Virginia University（西弗吉尼亚大学）；University at Buffalo, State University of New York（纽约州立大学布法罗分校）；The Hong Kong University of Science and Technology（香港科技大学）；Tongji University（同济大学）
We present an approach towards robust lane tracking for assisted and autonomous driving, particularly under poor visibility. Autonomous detection of lane markers improves road safety, and purely visual tracking is desirable for widespread vehicle compatibility and reducing sensor intrusion, cost, and energy consumption. However, visual approaches are often ineffective because of a number of factors, including but not limited to occlusion, poor weather conditions, and paint wear-off. Our method, named SafeDrive, attempts to improve visual lane detection approaches in drastically degraded visual…
For change detection in remote sensing, constructing a training dataset for deep learning models is difficult due to the requirements of bi-temporal supervision. To overcome this issue, single-temporal supervision which treats change labels as the difference of two semantic masks has been proposed. This novel method trains a change detector using two spatially unrelated images with corresponding semantic labels such as building. However, training on unpaired datasets could confuse the change detector in the case of pixels that are labeled unchanged but are visually significantly different. In …
Stanford University（斯坦福大学）；Renmin University of China（中国人民大学）
Understanding intrinsic patterns and predicting spatiotemporal characteristics of cities require a comprehensive representation of urban neighborhoods. Existing works relied on either inter- or intra-region connectivities to generate neighborhood representations but failed to fully utilize the informative yet heterogeneous data within neighborhoods. In this work, we propose Urban2Vec, an unsupervised multi-modal framework which incorporates both street view imagery and point-of-interest (POI) data to learn neighborhood embeddings. Specifically, we use a convolutional neural network to extract …
ZenSVI: An Open-Source Software for the Integrated Acquisition, Processing and Analysis of Street View Imagery Towards Scalable Urban Science
National University of Singapore（新加坡国立大学）；College of Big Data and Internet, Shenzhen Technology University（深圳技术大学大数据与互联网学院）；City University of Hong Kong（香港城市大学）；Wuhan University（武汉大学）；Singapore-ETH Centre（新加坡-ETH中心）
European Commission, Joint Research Centre (JRC)（欧盟委员会联合研究中心）
Street-level imagery holds a significant potential to scale-up in-situ data collection. This is enabled by combining the use of cheap high quality cameras with recent advances in deep learning compute solutions to derive relevant thematic information. We present a framework to collect and extract crop type and phenological information from street level imagery using computer vision. During the 2018 growing season, high definition pictures were captured with side-looking action cameras in the Flevoland province of the Netherlands. Each month from March to October, a fixed 200-km route was surve…
