We present a novel method for synthesizing both temporally and geometrically consistent street-view panoramic video from a single satellite image and camera trajectory. Existing cross-view synthesis approaches focus on images, while video synthesis in such a case has not yet received enough attention. For geometrical and temporal consistency, our approach explicitly creates a 3D point cloud representation of the scene and maintains dense 3D-2D correspondences across frames that reflect the geometric scene configuration inferred from the satellite view. As for synthesis in the 3D space, we implement a cascaded network architecture with two hourglass modules to generate point-wise coarse and fine features from semantics and per-class latent vectors, followed by projection to frames and an upsampling module to obtain the final realistic video. By leveraging computed correspondences, the produced street-view video frames adhere to the 3D geometric scene structure and maintain temporal consistency. Qualitative and quantitative experiments demonstrate superior results compared to other state-of-the-art synthesis approaches that either lack temporal consistency or realistic appearance. To the best of our knowledge, our work is the first one to synthesize cross-view images to video.
The goal of cross-view image based geo-localization is to determine the location of a given street view image by matching it against a collection of geo-tagged satellite images. This task is notoriously challenging due to the drastic viewpoint and appearance differences between the two domains. We show that we can address this discrepancy explicitly by learning to synthesize realistic street views from satellite inputs. Following this observation, we propose a novel multi-task architecture in which image synthesis and retrieval are considered jointly. The rationale behind this is that we can b…
Australian National University（澳大利亚国立大学）；University of Oxford（牛津大学）；University of Technology Sydney（悉尼科技大学）
This paper presents a new approach for synthesizing a novel street-view panorama given an overhead satellite image. Taking a small satellite image patch as input, our method generates a Google's omnidirectional street-view type panorama, as if it is captured from the same geographical location as the center of the satellite patch. Existing works tackle this task as an image generation problem which adopts generative adversarial networks to implicitly learn the cross-view transformations, while ignoring the domain relevance. In this paper, we propose to explicitly establish the geometric corres…
From Satellite to Street: A Hybrid Framework Integrating Stable Diffusion and PanoGAN for Consistent Cross-View Synthesis
Department of Information and Computer Science, King Fahd University of Petroleum and Minerals（信息与计算机科学系，国王法赫德石油与矿物大学）；NCMI, CSIRO（CSIRO国家科学机构）；Department of Computer Science, National University of Computer and Emerging Sciences (FAST-NUCES)（计算机科学系，国家计算机与新兴科学大学（FAST-NUCES））；Faculty of Engineering and IT, University of Technology Sydney（工程与信息技术学院，技术大学悉尼）；Faculty of Science, University of Ontario Institute of Technology（科学学院， Ontario Institute of Technology大学）
针对卫星到街景跨视角合成中的视角与外观差异，本文提出融合 Stable Diffusion、条件 GAN 与双分支多阶段训练的混合框架，在 CVUSA 上提升几何一致性与视觉真实感。
Hallucinating 360°: Panoramic Street-View Generation via Local Scenes Diffusion and Probabilistic Prompting
School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（人工智能与机器人学院和机器人视觉感知与控制技术国家工程研究中心，湖南大学）；Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology（人机学与机器人研究所，卡尔斯鲁厄技术大学）
Accurately maintaining digital street maps is labor-intensive. To address this challenge, much work has studied automatically processing geospatial data sources such as GPS trajectories and satellite images to reduce the cost of maintaining digital maps. An end-to-end map update system would first process geospatial data sources to extract insights, and second leverage those insights to update and improve the map. However, prior work largely focuses on the first step of this pipeline: these map extraction methods infer road networks from scratch given geospatial data sources (in effect creatin…
Space2Ground 2.0: A Multi-Source Dataset and Framework for Agricultural Monitoring through Fusion of Street-Level and Satellite Imagery
Operational Unit BEYOND Centre, IAASARS, National Observatory of Athens（雅典国家天文台IAASARS研究所BEYOND中心业务单元）；DHI Water & Environment（DHI水环境公司）；Artificial Intelligence Group, Wageningen University & Research（瓦赫宁根大学及研究中心人工智能组）
In this paper, based on the assumption that the object boundaries (e.g., buildings) from the over-view data should coincide with footprints of façade 3D points generated from street-view photogrammetric images, we aim to address this problem by proposing a fully automated geo-registration method for cross-view data, which utilizes semantically segmented object boundaries as view-invariant features under a global optimization framework through graph-matching: taking the over-view point clouds generated from stereo/multi-stereo satellite images and the street-view point clouds generated from mon…
IRISA institute, Université Bretagne Sud（IRISA研究所，南布列塔尼大学）；MultiModal Remote Sensing group, University of Zurich（苏黎世大学多模态遥感研究组）；Laboratory of GeoInformation Science and Remote Sensing, Wageningen University & Research（瓦赫宁根大学及研究中心地理信息科学与遥感实验室）；Photogrammetry and Remote Sensing group, ETH Zurich（苏黎世联邦理工学院摄影测量与遥感研究组）；G2C institute, University of Applied Sciences, Western Switzerland（瑞士西部应用科学与艺术大学G2C研究所）
In this paper, we discuss and review how combined multi-view imagery from satellite to street-level can benefit scene analysis. Numerous works exist that merge information from remote sensing and images acquired from the ground for tasks like land cover mapping, object detection, or scene understanding. What makes the combination of overhead and street-level images challenging, is the strongly varying viewpoint, different scale, illumination, sensor modality and time of acquisition. Direct (dense) matching of images on a per-pixel basis is thus often impossible, and one has to resort to altern…
How to find places that are not indexed by Google Maps? We propose an intuitive method and framework to locate places based on their distinctive spatial features. The method uses satellite and street view images in machine vision approaches to classify locations. If we can classify locations, we just need to repeat for non-overlapping locations in our area of interest. We assess the proposed system in finding Parkour spots in the campus of Arizona State University. The results are very satisfactory, having found more than 25 new Parkour spots, with a rate of true positives above 60%.
Beyond Geo-localization: Fine-grained Orientation of Street-view Images by Cross-view Matching with Satellite Imagery with Supplementary Materials
National University of Singapore（新加坡国立大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Grab Chronos S.R.L.（Grab Chronos有限责任公司）；Grabtaxi Holdings Pte. Ltd.（Grab出租车控股私人有限公司）
Multimodal Urban Tree Detection from Satellite and Street-Level Imagery via Annotation-Efficient Deep Learning Strategies
Department of Computer Science, University of California, Davis（加州大学戴维斯分校计算机科学系）；Department of Biological and Agricultural Engineering, University of California, Davis（加州大学戴维斯分校生物与农业工程系）
State Key Laboratory of Optical Instrumentation, Zhejiang University, China（浙江大学光学仪器国家重点实验室）；Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany（卡尔斯鲁厄理工学院人机与机器人研究所）
Aerial pixel-wise scene perception of the surrounding environment is an important task for UAVs (Unmanned Aerial Vehicles). Previous research works mainly adopt conventional pinhole cameras or fisheye cameras as the imaging device. However, these imaging systems cannot achieve large Field of View (FoV), small size, and lightweight at the same time. To this end, we design a UAV system with a Panoramic Annular Lens (PAL), which has the characteristics of small size, low weight, and a 360-degree annular FoV. A lightweight panoramic annular semantic segmentation neural network model is designed to…
Cross-View Geo-Localization with Street-View and VHR Satellite Imagery in Decentrality Settings
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；Ant Group（蚂蚁集团）；Technology Innovation Center for Collaborative Applications of Natural Resources Data in GBA, Ministry of Natural Resources（自然资源部粤港澳自然资源数据协同应用技术创新中心）
Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）；Vellore Institute of Technology（韦洛尔理工学院）；College of Engineering, Anna University（安娜大学工程学院）；Measi Academy Of Architecture（米西建筑学院）
Federated Deep Learning frameworks can be used strategically to monitor Land Use locally and infer environmental impacts globally. Distributed data from across the world would be needed to build a global model for Land Use classification. The need for a Federated approach in this application domain would be to avoid transfer of data from distributed locations and save network bandwidth to reduce communication cost. We use a Federated UNet model for Semantic Segmentation of satellite and street view images. The novelty of the proposed architecture is the integration of Knowledge Distillation to…
SkyLink: Unifying Street-Satellite Geo-Localization via UAV-Mediated 3D Scene Alignment
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，教育部，厦门大学）；School of Electronic and Information Engineering, Foshan University（电子与信息工程学院，佛山大学）
CarbonCLIP: Enhance Carbon Prediction from Satellite Imagery via Integrated Street-View Semantics and Temporal Context Training
Energy Research Institute at NTU（国立理工学院能源研究所）；Interdisciplinary Graduate Programme（跨学科研究生项目）；School of Electrical and Electronic Engineering（电气电子工程学院）；School of Public Administration and Policy（公共管理与政策学院）；Asian School of the Environment（亚洲环境学院）；Center for Climate Change and Environmental Health（气候变化与环境健康中心）
Recov-Vision: Linking Street View Imagery and Vision-Language Models for Post-Disaster Recovery
UrbanResilience.AI Lab（UrbanResilience.AI 实验室）；Zachry Department of Civil and Environmental Engineering（Zachry 土木与环境工程系）；Texas A&M University（德克萨斯A&M大学）；Department of Earth, Marine and Environmental Sciences（地球、海洋与环境科学系）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Location recognition is commonly treated as visual instance retrieval on "street view" imagery. The dataset items and queries are panoramic views, i.e. groups of images taken at a single location. This work introduces a novel panorama-to-panorama matching process, either by aggregating features of individual images in a group or by explicitly constructing a larger panorama. In either case, multiple views are used as queries. We reach near perfect location recognition on a standard benchmark with only four query views.
From Street to Orbit: Training-Free Cross-View Retrieval via Location Semantics and LLM Guidance
Ulsan National Institute of Science and Technology (UNIST)（蔚山科学技术院）；Electronics and Telecommunications Research Institute (ETRI)（电子通信研究院）；Kyungpook National University（庆北大学）
Massachusetts Institute of Technology（麻省理工学院）；Draper（德雷珀实验室）
Cross-view image geolocalization provides an estimate of an agent's global position by matching a local ground image to an overhead satellite image without the need for GPS. It is challenging to reliably match a ground image to the correct satellite image since the images have significant viewpoint differences. Existing works have demonstrated localization in constrained scenarios over small areas but have not demonstrated wider-scale localization. Our approach, called Wide-Area Geolocalization (WAG), combines a neural network with a particle filter to achieve global position estimates for age…
Alan Turing Institute（艾伦·图灵研究所）；University College London（伦敦大学学院）；University of Cambridge（剑桥大学）；University of Surrey（萨里大学）
When an individual purchases a home, they simultaneously purchase its structural features, its accessibility to work, and the neighborhood amenities. Some amenities, such as air quality, are measurable while others, such as the prestige or the visual impression of a neighborhood, are difficult to quantify. Despite the well-known impacts intangible housing features have on house prices, limited attention has been given to systematically quantifying these difficult to measure amenities. Two issues have led to this neglect. Not only do few quantitative methods exist that can measure the urban env…
Engineering Vellore Institute of Technology, AP Guntur, India（工程维洛雷理工学院，AP古尔图，印度）；Engineering Vellore Institute of Technology, AP Ongole, India（工程维洛雷理工学院，AP翁戈尔，印度）；Engineering Vellore Institute of Technology, AP Vijayawada, India（工程维洛雷理工学院，AP维杰亚瓦达，印度）
We present a novel approach to geolocalising panoramic images on a 2-D cartographic map based on learning a low dimensional embedded space, which allows a comparison between an image captured at a location and local neighbourhoods of the map. The representation is not sufficiently discriminatory to allow localisation from a single image, but when concatenated along a route, localisation converges quickly, with over 90% accuracy being achieved for routes of around 200m in length when using Google Street View and Open Street Map data. The method generalises a previous fixed semantic feature base…
OpenFACADES: An Open Framework for Architectural Caption and Attribute Data Enrichment via Street View Imagery
Department of Architecture, National University of Singapore（建筑系，新加坡国立大学）；Department of Electrical and Computer Engineering, National University of Singapore（电气与计算机工程系，新加坡国立大学）；School of Artificial Intelligence, Shenzhen Technology University（人工智能学院，深圳科技大学）；Department of Real Estate, National University of Singapore（房地产系，新加坡国立大学）
SA-Occ: Satellite-Assisted 3D Occupancy Prediction in Real World
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
Sun Yat-sen University（中山大学）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Wuhan University（武汉大学）
This paper presents OmniCity, a new dataset for omnipotent city understanding from multi-level and multi-view images. More precisely, the OmniCity contains multi-view satellite images as well as street-level panorama and mono-view images, constituting over 100K pixel-wise annotated images that are well-aligned and collected from 25K geo-locations in New York City. To alleviate the substantial pixel-wise annotation efforts, we propose an efficient street-view image annotation pipeline that leverages the existing label maps of satellite view and the transformation relations between different vie…
Department of Information and Computer Science（信息与计算机科学系）；King Fahd University of Petroleum and Minerals（国王法赫德石油与矿物大学）；SDAIA-KFUPM Joint Research Center of Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）；Ontario Tech University（安大略技术大学）；University of Canberra（堪培拉大学）；University of Regina（里贾纳大学）
In this paper we aim to determine the location and orientation of a ground-level query image by matching to a reference database of overhead (e.g. satellite) images. For this task we collect a new dataset with one million pairs of street view and overhead images sampled from eleven U.S. cities. We explore several deep CNN architectures for cross-domain matching -- Classification, Hybrid, Siamese, and Triplet networks. Classification and Hybrid architectures are accurate but slow since they allow only partial feature precomputation. We propose a new loss function which significantly improves th…
Cross-View Geolocalization and Disaster Mapping with Street-View and VHR Satellite Imagery: A Case Study of Hurricane IAN
Technical University of Munich（慕尼黑工业大学）；University of the Bundeswehr Munich（慕尼黑联邦国防军大学）；China University of Mining and Technology（中国矿业大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Tongji University（同济大学）
PetalView: Fine-grained Location and Orientation Extraction of Street-view Images via Cross-view Local Search with Supplementary Materials
Grab-NUS AI Lab（Grab-NUS人工智能实验室）；National University of Singapore（新加坡国立大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute for Infocomm Research, A*STAR（新加坡资讯通信研究院（新加坡科技研究局下属））；Grabtaxi Holdings Pte. Ltd.（GrabTaxi控股私人有限公司）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究院）
Large collections of geo-referenced panoramic images are freely available for cities across the globe, as well as detailed maps with location and meta-data on a great variety of urban objects. They provide a potentially rich source of information on urban objects, but manual annotation for object detection is costly, laborious and difficult. Can we utilize such multimedia sources to automatically annotate street level images as an inexpensive alternative to manual labeling? With the PanorAMS framework we introduce a method to automatically generate bounding box annotations for panoramic images…
AddressVLM: Cross-view Alignment Tuning for Image Address Localization using Large Vision-Language Models
State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；CASIA（中国科学院自动化所）；Alibaba Cloud（阿里云）；School of Intelligence Science and Technology（智能科学与技术学院）；CAIR（中国科学院香港创新研究院）
Institute of Space Technology（空间技术研究所）；University of St Andrews（圣安德鲁斯大学）；Univ. Gustave Eiffel（古斯塔夫·埃菲尔大学）；IGN（法国国家地理与林业信息研究所）；ENSG（法国国立地理科学学院）；JIS University（JIS大学）
The Corona KH-4 reconnaissance satellite missions from 1962-1972 acquired panoramic stereo imagery with high spatial resolution of 1.8-7.5 m. The potential of 800,000+ declassified Corona images has not been leveraged due to the complexities arising from handling of panoramic imaging geometry, film distortions and limited availability of the metadata required for georeferencing of the Corona imagery. This paper presents Corona Stereo Pipeline (CoSP): A pipeline for processing of Corona KH-4 stereo panoramic imagery. CoSP utlizes a deep learning based feature matcher SuperGlue to automatically …
University of Melbourne（墨尔本大学）；University of the Sunshine Coast（阳光海岸大学）
The confluence of recent advances in availability of geospatial information, computing power, and artificial intelligence offers new opportunities to understand how and where our cities differ or are alike. Departing from a traditional `top-down' analysis of urban design features, this project analyses millions of images of urban form (consisting of street view, satellite imagery, and street maps) to find shared characteristics. A (novel) neural network-based framework is trained with imagery from the largest 1692 cities in the world and the resulting models are used to compare within-city loc…
