Satellite-to-street view synthesis aims at generating a realistic street-view image from its corresponding satellite-view image. Although stable diffusion models have exhibit remarkable performance in a variety of image generation applications, their reliance on similar-view inputs to control the generated structure or texture restricts their application to the challenging cross-view synthesis task. In this work, we propose CrossViewDiff, a cross-view diffusion model for satellite-to-street view synthesis. To address the challenges posed by the large discrepancy across views, we design the satellite scene structure estimation and cross-view texture mapping modules to construct the structural and textural controls for street-view image synthesis. We further design a cross-view control guided denoising process that incorporates the above controls via an enhanced cross-view attention module. To achieve a more comprehensive evaluation of the synthesis results, we additionally design a GPT-based scoring method as a supplement to standard evaluation metrics. We also explore the effect of different data sources (e.g., text, maps, building heights, and multi-temporal satellite imagery) on this task. Results on three public cross-view datasets show that CrossViewDiff outperforms current state-of-the-art on both standard and GPT-based evaluation metrics, generating high-quality street-view panoramas with more realistic structures and textures across rural, suburban, and urban scenes. The code and models of this work will be released at https://opendatalab.github.io/CrossViewDiff/.
The goal of cross-view image based geo-localization is to determine the location of a given street view image by matching it against a collection of geo-tagged satellite images. This task is notoriously challenging due to the drastic viewpoint and appearance differences between the two domains. We show that we can address this discrepancy explicitly by learning to synthesize realistic street views from satellite inputs. Following this observation, we propose a novel multi-task architecture in which image synthesis and retrieval are considered jointly. The rationale behind this is that we can b…
Australian National University（澳大利亚国立大学）；University of Oxford（牛津大学）；University of Technology Sydney（悉尼科技大学）
This paper presents a new approach for synthesizing a novel street-view panorama given an overhead satellite image. Taking a small satellite image patch as input, our method generates a Google's omnidirectional street-view type panorama, as if it is captured from the same geographical location as the center of the satellite patch. Existing works tackle this task as an image generation problem which adopts generative adversarial networks to implicitly learn the cross-view transformations, while ignoring the domain relevance. In this paper, we propose to explicitly establish the geometric corres…
ETH Zurich（苏黎世联邦理工学院）；The University of Tokyo（东京大学）；Zhejiang University（浙江大学）；The Ohio State University（俄亥俄州立大学）；Microsoft（微软）
We present a novel method for synthesizing both temporally and geometrically consistent street-view panoramic video from a single satellite image and camera trajectory. Existing cross-view synthesis approaches focus on images, while video synthesis in such a case has not yet received enough attention. For geometrical and temporal consistency, our approach explicitly creates a 3D point cloud representation of the scene and maintains dense 3D-2D correspondences across frames that reflect the geometric scene configuration inferred from the satellite view. As for synthesis in the 3D space, we impl…
From Satellite to Street: A Hybrid Framework Integrating Stable Diffusion and PanoGAN for Consistent Cross-View Synthesis
Department of Information and Computer Science, King Fahd University of Petroleum and Minerals（信息与计算机科学系，国王法赫德石油与矿物大学）；NCMI, CSIRO（CSIRO国家科学机构）；Department of Computer Science, National University of Computer and Emerging Sciences (FAST-NUCES)（计算机科学系，国家计算机与新兴科学大学（FAST-NUCES））；Faculty of Engineering and IT, University of Technology Sydney（工程与信息技术学院，技术大学悉尼）；Faculty of Science, University of Ontario Institute of Technology（科学学院， Ontario Institute of Technology大学）
针对卫星到街景跨视角合成中的视角与外观差异，本文提出融合 Stable Diffusion、条件 GAN 与双分支多阶段训练的混合框架，在 CVUSA 上提升几何一致性与视觉真实感。
Accurately maintaining digital street maps is labor-intensive. To address this challenge, much work has studied automatically processing geospatial data sources such as GPS trajectories and satellite images to reduce the cost of maintaining digital maps. An end-to-end map update system would first process geospatial data sources to extract insights, and second leverage those insights to update and improve the map. However, prior work largely focuses on the first step of this pipeline: these map extraction methods infer road networks from scratch given geospatial data sources (in effect creatin…
Space2Ground 2.0: A Multi-Source Dataset and Framework for Agricultural Monitoring through Fusion of Street-Level and Satellite Imagery
Operational Unit BEYOND Centre, IAASARS, National Observatory of Athens（雅典国家天文台IAASARS研究所BEYOND中心业务单元）；DHI Water & Environment（DHI水环境公司）；Artificial Intelligence Group, Wageningen University & Research（瓦赫宁根大学及研究中心人工智能组）
How to find places that are not indexed by Google Maps? We propose an intuitive method and framework to locate places based on their distinctive spatial features. The method uses satellite and street view images in machine vision approaches to classify locations. If we can classify locations, we just need to repeat for non-overlapping locations in our area of interest. We assess the proposed system in finding Parkour spots in the campus of Arizona State University. The results are very satisfactory, having found more than 25 new Parkour spots, with a rate of true positives above 60%.
In this paper, based on the assumption that the object boundaries (e.g., buildings) from the over-view data should coincide with footprints of façade 3D points generated from street-view photogrammetric images, we aim to address this problem by proposing a fully automated geo-registration method for cross-view data, which utilizes semantically segmented object boundaries as view-invariant features under a global optimization framework through graph-matching: taking the over-view point clouds generated from stereo/multi-stereo satellite images and the street-view point clouds generated from mon…
IRISA institute, Université Bretagne Sud（IRISA研究所，南布列塔尼大学）；MultiModal Remote Sensing group, University of Zurich（苏黎世大学多模态遥感研究组）；Laboratory of GeoInformation Science and Remote Sensing, Wageningen University & Research（瓦赫宁根大学及研究中心地理信息科学与遥感实验室）；Photogrammetry and Remote Sensing group, ETH Zurich（苏黎世联邦理工学院摄影测量与遥感研究组）；G2C institute, University of Applied Sciences, Western Switzerland（瑞士西部应用科学与艺术大学G2C研究所）
In this paper, we discuss and review how combined multi-view imagery from satellite to street-level can benefit scene analysis. Numerous works exist that merge information from remote sensing and images acquired from the ground for tasks like land cover mapping, object detection, or scene understanding. What makes the combination of overhead and street-level images challenging, is the strongly varying viewpoint, different scale, illumination, sensor modality and time of acquisition. Direct (dense) matching of images on a per-pixel basis is thus often impossible, and one has to resort to altern…
Beyond Geo-localization: Fine-grained Orientation of Street-view Images by Cross-view Matching with Satellite Imagery with Supplementary Materials
National University of Singapore（新加坡国立大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Grab Chronos S.R.L.（Grab Chronos有限责任公司）；Grabtaxi Holdings Pte. Ltd.（Grab出租车控股私人有限公司）
Multimodal Urban Tree Detection from Satellite and Street-Level Imagery via Annotation-Efficient Deep Learning Strategies
Department of Computer Science, University of California, Davis（加州大学戴维斯分校计算机科学系）；Department of Biological and Agricultural Engineering, University of California, Davis（加州大学戴维斯分校生物与农业工程系）
Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）；Vellore Institute of Technology（韦洛尔理工学院）；College of Engineering, Anna University（安娜大学工程学院）；Measi Academy Of Architecture（米西建筑学院）
Federated Deep Learning frameworks can be used strategically to monitor Land Use locally and infer environmental impacts globally. Distributed data from across the world would be needed to build a global model for Land Use classification. The need for a Federated approach in this application domain would be to avoid transfer of data from distributed locations and save network bandwidth to reduce communication cost. We use a Federated UNet model for Semantic Segmentation of satellite and street view images. The novelty of the proposed architecture is the integration of Knowledge Distillation to…
SkyLink: Unifying Street-Satellite Geo-Localization via UAV-Mediated 3D Scene Alignment
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，教育部，厦门大学）；School of Electronic and Information Engineering, Foshan University（电子与信息工程学院，佛山大学）
Cross-View Geo-Localization with Street-View and VHR Satellite Imagery in Decentrality Settings
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；Ant Group（蚂蚁集团）；Technology Innovation Center for Collaborative Applications of Natural Resources Data in GBA, Ministry of Natural Resources（自然资源部粤港澳自然资源数据协同应用技术创新中心）
CarbonCLIP: Enhance Carbon Prediction from Satellite Imagery via Integrated Street-View Semantics and Temporal Context Training
Energy Research Institute at NTU（国立理工学院能源研究所）；Interdisciplinary Graduate Programme（跨学科研究生项目）；School of Electrical and Electronic Engineering（电气电子工程学院）；School of Public Administration and Policy（公共管理与政策学院）；Asian School of the Environment（亚洲环境学院）；Center for Climate Change and Environmental Health（气候变化与环境健康中心）
From Street to Orbit: Training-Free Cross-View Retrieval via Location Semantics and LLM Guidance
Ulsan National Institute of Science and Technology (UNIST)（蔚山科学技术院）；Electronics and Telecommunications Research Institute (ETRI)（电子通信研究院）；Kyungpook National University（庆北大学）
Massachusetts Institute of Technology（麻省理工学院）；Draper（德雷珀实验室）
Cross-view image geolocalization provides an estimate of an agent's global position by matching a local ground image to an overhead satellite image without the need for GPS. It is challenging to reliably match a ground image to the correct satellite image since the images have significant viewpoint differences. Existing works have demonstrated localization in constrained scenarios over small areas but have not demonstrated wider-scale localization. Our approach, called Wide-Area Geolocalization (WAG), combines a neural network with a particle filter to achieve global position estimates for age…
Alan Turing Institute（艾伦·图灵研究所）；University College London（伦敦大学学院）；University of Cambridge（剑桥大学）；University of Surrey（萨里大学）
When an individual purchases a home, they simultaneously purchase its structural features, its accessibility to work, and the neighborhood amenities. Some amenities, such as air quality, are measurable while others, such as the prestige or the visual impression of a neighborhood, are difficult to quantify. Despite the well-known impacts intangible housing features have on house prices, limited attention has been given to systematically quantifying these difficult to measure amenities. Two issues have led to this neglect. Not only do few quantitative methods exist that can measure the urban env…
Engineering Vellore Institute of Technology, AP Guntur, India（工程维洛雷理工学院，AP古尔图，印度）；Engineering Vellore Institute of Technology, AP Ongole, India（工程维洛雷理工学院，AP翁戈尔，印度）；Engineering Vellore Institute of Technology, AP Vijayawada, India（工程维洛雷理工学院，AP维杰亚瓦达，印度）
SA-Occ: Satellite-Assisted 3D Occupancy Prediction in Real World
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
Department of Information and Computer Science（信息与计算机科学系）；King Fahd University of Petroleum and Minerals（国王法赫德石油与矿物大学）；SDAIA-KFUPM Joint Research Center of Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）；Ontario Tech University（安大略技术大学）；University of Canberra（堪培拉大学）；University of Regina（里贾纳大学）
Sun Yat-sen University（中山大学）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Wuhan University（武汉大学）
This paper presents OmniCity, a new dataset for omnipotent city understanding from multi-level and multi-view images. More precisely, the OmniCity contains multi-view satellite images as well as street-level panorama and mono-view images, constituting over 100K pixel-wise annotated images that are well-aligned and collected from 25K geo-locations in New York City. To alleviate the substantial pixel-wise annotation efforts, we propose an efficient street-view image annotation pipeline that leverages the existing label maps of satellite view and the transformation relations between different vie…
Cross-View Geolocalization and Disaster Mapping with Street-View and VHR Satellite Imagery: A Case Study of Hurricane IAN
Technical University of Munich（慕尼黑工业大学）；University of the Bundeswehr Munich（慕尼黑联邦国防军大学）；China University of Mining and Technology（中国矿业大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Tongji University（同济大学）
In this paper we aim to determine the location and orientation of a ground-level query image by matching to a reference database of overhead (e.g. satellite) images. For this task we collect a new dataset with one million pairs of street view and overhead images sampled from eleven U.S. cities. We explore several deep CNN architectures for cross-domain matching -- Classification, Hybrid, Siamese, and Triplet networks. Classification and Hybrid architectures are accurate but slow since they allow only partial feature precomputation. We propose a new loss function which significantly improves th…
PetalView: Fine-grained Location and Orientation Extraction of Street-view Images via Cross-view Local Search with Supplementary Materials
Grab-NUS AI Lab（Grab-NUS人工智能实验室）；National University of Singapore（新加坡国立大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute for Infocomm Research, A*STAR（新加坡资讯通信研究院（新加坡科技研究局下属））；Grabtaxi Holdings Pte. Ltd.（GrabTaxi控股私人有限公司）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究院）
AddressVLM: Cross-view Alignment Tuning for Image Address Localization using Large Vision-Language Models
State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；CASIA（中国科学院自动化所）；Alibaba Cloud（阿里云）；School of Intelligence Science and Technology（智能科学与技术学院）；CAIR（中国科学院香港创新研究院）
University of Melbourne（墨尔本大学）；University of the Sunshine Coast（阳光海岸大学）
The confluence of recent advances in availability of geospatial information, computing power, and artificial intelligence offers new opportunities to understand how and where our cities differ or are alike. Departing from a traditional `top-down' analysis of urban design features, this project analyses millions of images of urban form (consisting of street view, satellite imagery, and street maps) to find shared characteristics. A (novel) neural network-based framework is trained with imagery from the largest 1692 cities in the world and the resulting models are used to compare within-city loc…
Accurate information about the location and orientation of a camera in mobile devices is central to the utilization of location-based services (LBS). Most of such mobile devices rely on GPS data but this data is subject to inaccuracy due to imperfections in the quality of the signal provided by satellites. This shortcoming has spurred the research into improving the accuracy of localization. Since mobile devices have camera, a major thrust of this research has been seeks to acquire the local scene and apply image retrieval techniques by querying a GPS-tagged image database to find the best mat…
Stanford University（斯坦福大学）；IBM Research Europe（IBM欧洲研究院）；IBM Europe（IBM欧洲）
Current methods to determine the energy efficiency of buildings require on-site visits of certified energy auditors which makes the process slow, costly, and geographically incomplete. To accelerate the identification of promising retrofit targets on a large scale, we propose to estimate building energy efficiency from widely available and remotely sensed data sources only, namely street view, aerial view, footprint, and satellite-borne land surface temperature (LST) data. After collecting data for almost 40,000 buildings in the United Kingdom, we combine these data sources by training multipl…
Sat2City v2: Native 3D City Asset Generation from a Single Satellite Image
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Chinese University of Hong Kong（香港中文大学）；Auckland University of Technology（奥克兰理工大学）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
CVSBench: A Comprehensive Benchmark for Cross-view Spatial Reasoning and Dreaming
Faculty of Electronic and Information Engineering, Xi’an Jiaotong University（西安交通大学电子与信息工程学院）；Ministry of Education Key Laboratory of Intelligent Networks and Network Security（教育部智能网络与网络安全重点实验室）
QuantCube Technology（QuantCube科技公司）；Univ. Bretagne Sud（南布列塔尼大学）；Centre National d'Etudes Spatiales（法国国家空间研究中心）
Detection of new infrastructures (commercial, logistics, industrial or residential) from satellite images constitutes a proven method to investigate and follow economic and urban growth. The level of activities or exploitation of these sites may be hardly determined by building inspection, but could be inferred from vehicle presence from nearby streets and parking lots. We present in this paper two deep learning-based models for vehicle counting from optical satellite images coming from the Pleiades sensor at 50-cm spatial resolution. Both segmentation (Tiramisu) and detection (YOLO) architect…
Lahore University of Management Sciences（拉合尔管理科学大学）
Cross-modal retrieval aims to measure the content similarity between different types of data. The idea has been previously applied to visual, text, and speech data. In this paper, we present a novel cross-modal retrieval method specifically for multi-view images, called Cross-view Image Retrieval CVIR. Our approach aims to find a feature space as well as an embedding space in which samples from street-view images are compared directly to satellite-view images (and vice-versa). For this comparison, a novel deep metric learning based solution "DeepCVIR" has been proposed. Previous cross-view ima…
UB-FineNet: Urban Building Fine-grained Classification Network for Open-access Satellite Images
The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong Polytechnic University Shenzhen Research Institute（香港理工大学深圳研究院）；Otto Poon Charitable Foundation Smart Cities Research Institute, The Hong Kong Polytechnic University（香港理工大学潘奥本慈善基金会智慧城市研究院）
Geisel School of Medicine at Dartmouth（达特茅斯盖泽尔医学院）；Dartmouth Hitchcock Medical Center（达特茅斯-希区柯克医疗中心）；The Dartmouth Institute for Health Policy and Clinical Practice（达特茅斯健康政策与临床实践研究所）
Importance: Following a century of increase, life expectancy in the United States has stagnated and begun to decline in recent decades. Using satellite images and street view images prior work has demonstrated associations of the built environment with income, education, access to care and health factors such as obesity. However, assessment of learned image feature relationships with variation in crude mortality rate across the United States has been lacking. Objective: Investigate prediction of county-level mortality rates in the U.S. using satellite images. Design: Satellite images were extr…
Multi-View Mixture-of-Experts with Vision-Language Reranking for Cross-View Object Geo-Localization
College of Computer Science, Beijing University of Technology（北京工业大学计算机科学学院）；Zhengzhou University（郑州大学）；Zhejiang University（浙江大学）；Beijing Institute of Technology（北京理工大学）；Beijing Electronic Science and Technology Institute（北京电子科技学院）；Intelligent Science & Technology Academy of CASIC（中国航天科工集团智能科学技术研究院）
GeoBridge++: Fact-Guided Geo-Semantic Bridging for Unified Cross-View Geo-Localization
School of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
