Fusion and Grouping Strategies in Deep Learning for Local Climate Zone Classification of Multimodal Remote Sensing Data
Graphics-Visualization-Computing Lab, International Institute of Information Technology Bangalore, Karnataka 560100, India（图形可视化计算实验室，国际信息学院班加罗尔，卡纳塔克邦560100，印度）
Local Climate Zones (LCZs) give a zoning map to study urban structures and land use and analyze the impact of urbanization on local climate. Multimodal remote sensing enables LCZ classification, for which data fusion is significant for improving accuracy owing to the data complexity. However, there is a gap in a comprehensive analysis of the fusion mechanisms used in their deep learning (DL) classifier architectures. This study analyzes different fusion strategies in the multi-class LCZ classification models for multimodal data and grouping strategies based on inherent data characteristics. The different models involving Convolutional Neural Networks (CNNs) include: (i) baseline hybrid fusion (FM1), (ii) with self- and cross-attention mechanisms (FM2), (iii) with the multi-scale Gaussian filtered images (FM3), and (iv) weighted decision-level fusion (FM4). Ablation experiments are conducted to study the pixel-, feature-, and decision-level fusion effects in the model performance. Grouping strategies include band grouping (BG) within the data modalities and label merging (LM) in the ground truth. Our analysis is exclusively done on the So2Sat LCZ42 dataset, which consists of Synthetic Aperture Radar (SAR) and Multispectral Imaging (MSI) image pairs. Our results show that FM1 consistently outperforms simple fusion methods. FM1 with BG and LM is found to be the most effective approach among all fusion strategies, giving an overall accuracy of 76.6\%. Importantly, our study highlights the effect of these strategies in improving prediction accuracy for the underrepresented classes. Our code and processed datasets are available at https://github.com/GVCL/LCZC-MultiModalHybridFusion
Predicting unobserved climate time series data at distant areas via spatial correlation using reservoir computing
Toyota Central R&D Labs., Inc.（丰田中央研发实验室）；International Research Center for Neurointelligence, The University of Tokyo（东京大学国际神经智能研究中心）；Graduate School of Engineering, Nagoya Institute of Technology（名古屋工业大学工学研究科）
Over the last few decades, deforestation and climate change have caused increasing number of forest fires. In Southeast Asia, Indonesia has been the most affected country by tropical peatland forest fires. These fires have a significant impact on the climate resulting in extensive health, social and economic issues. Existing forest fire prediction systems, such as the Canadian Forest Fire Danger Rating System, are based on handcrafted features and require installation and maintenance of expensive instruments on the ground, which can be a challenge for developing countries such as Indonesia. We…
Global Mapping of Exposure and Physical Vulnerability Dynamics in Least Developed Countries using Remote Sensing and Machine Learning
UKRI CDT in AI for Environmental Risks（英国研究与创新人工智能环境风险博士培训中心）；Centre for Risk in the Built Environment（建筑环境风险中心）；University of Cambridge（剑桥大学）；German Aerospace Center (DLR)（德国航空航天中心）；University of Bonn（波恩大学）
Air pollution is a major driver of climate change. Anthropogenic emissions from the burning of fossil fuels for transportation and power generation emit large amounts of problematic air pollutants, including Greenhouse Gases (GHGs). Despite the importance of limiting GHG emissions to mitigate climate change, detailed information about the spatial and temporal distribution of GHG and other air pollutants is difficult to obtain. Existing models for surface-level air pollution rely on extensive land-use datasets which are often locally restricted and temporally static. This work proposes a deep l…
AI-driven multi-source data fusion for algal bloom severity classification in small inland water bodies: Leveraging Sentinel-2, DEM, and NOAA climate data
The problem of supervised classification of the satellite image is considered to be the task of grouping pixels into a number of homogeneous regions in space intensity. This paper proposes a novel approach that combines a radial basic function clustering network with a growing neural gas include utility factor classifier to yield improved solutions, obtained with previous networks. The double objective technique is first used to the development of a method to perform the satellite images classification, and finally, the implementation to address the issue of the number of nodes in the hidden l…
DF4LCZ: A SAM-Empowered Data Fusion Framework for Scene-Level Local Climate Zone Classification
Future Network of Intelligence Institute (FNii), the Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）未来智能网络研究院（FNii））；School of Science and Engineering, the Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）
Optical remote sensing imagery has been widely used in many fields due to its high resolution and stable geometric properties. However, remote sensing imagery is inevitably affected by climate, especially clouds. Removing the cloud in the high-resolution remote sensing satellite image is an indispensable pre-processing step before analyzing it. For the sake of large-scale training data, neural networks have been successful in many image processing tasks, but the use of neural networks to remove cloud in remote sensing imagery is still relatively small. We adopt generative adversarial network t…
In many developing nations, a lack of poverty data prevents critical humanitarian organizations from responding to large-scale crises. Currently, socioeconomic surveys are the only method implemented on a large scale for organizations and researchers to measure and track poverty. However, the inability to collect survey data efficiently and inexpensively leads to significant temporal gaps in poverty data; these gaps severely limit the ability of organizational entities to address poverty at its root cause. We propose a transfer learning model based on surface temperature change and remote sens…
Element AI；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
Remote sensing and automatic earth monitoring are key to solve global-scale challenges such as disaster prevention, land use monitoring, or tackling climate change. Although there exist vast amounts of remote sensing data, most of it remains unlabeled and thus inaccessible for supervised learning algorithms. Transfer learning approaches can reduce the data requirements of deep learning algorithms. However, most of these methods are pre-trained on ImageNet and their generalization to remote sensing imagery is not guaranteed due to the domain gap. In this work, we propose Seasonal Contrast (SeCo…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft（微软）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Remotely sensed geospatial data are critical for applications including precision agriculture, urban planning, disaster monitoring and response, and climate change research, among others. Deep learning methods are particularly promising for modeling many remote sensing tasks given the success of deep neural networks in similar computer vision tasks and the sheer volume of remotely sensed imagery available. However, the variance in data collection methods and handling of geospatial metadata make the application of deep learning methodology to remotely sensed data nontrivial. For example, satell…
Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR)（德国航空航天中心（DLR）遥感技术研究所（IMF））；German Aerospace Center (DLR)（德国航空航天中心（DLR））；Signal Processing in Earth Observation (SiPEO), Technical University of Munich (TUM)（慕尼黑工业大学（TUM）地球观测信号处理实验室（SiPEO））；Technical University of Munich (TUM)（慕尼黑工业大学（TUM））
Access to labeled reference data is one of the grand challenges in supervised machine learning endeavors. This is especially true for an automated analysis of remote sensing images on a global scale, which enables us to address global challenges such as urbanization and climate change using state-of-the-art machine learning techniques. To meet these pressing needs, especially in urban research, we provide open access to a valuable benchmark dataset named "So2Sat LCZ42," which consists of local climate zone (LCZ) labels of about half a million Sentinel-1 and Sentinel-2 image patches in 42 urban…
Peking University（北京大学）；University of Cambridge（剑桥大学）
In this paper, we propose an efficient and generalizable framework based on deep convolutional neural network (CNN) for multi-source remote sensing data joint classification. While recent methods are mostly based on multi-stream architectures, we use group convolution to construct equivalent network architectures efficiently within a single-stream network. We further adopt and improve dynamic grouping convolution (DGConv) to make group convolution hyperparameters, and thus the overall network architecture, learnable during network training. The proposed method therefore can theoretically adjus…
Estimating forest carbon stocks from high-resolution remote sensing imagery by reducing domain shift with style transfer
School of Geography and Remote Sensing, Guangzhou University（广州大学地理与遥感学院）；Innovation Center for Remote Sensing Big Data Intelligent Applications, Guangzhou University（广州大学遥感大数据智能应用创新中心）
Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR)（德国航空航天中心遥感技术研究所）；Signal Processing in Earth Observation (SiPEO), Technical University of Munich (TUM)（慕尼黑工业大学地球观测信号处理实验室）；Department of Geography, University of Zurich（苏黎世大学地理系）；Laboratory of GeoInformation Science and Remote Sensing, Wageningen University of Research（瓦赫宁根大学地理信息科学与遥感实验室）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Institute of Computer Graphics and Vision, TU Graz（格拉茨技术大学计算机图形与视觉研究所）
Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensi…
Global-Local Detail Guided Transformer for Sea Ice Recognition in Optical Remote Sensing Images
The Academy of Digital China, Fuzhou University（福州大学数字中国研究院）；The Key Laboratory of Spatial Data Mining and Information Sharing of Ministry of Education（教育部空间数据挖掘与信息共享重点实验室）
Generating Physically-Consistent Satellite Imagery for Climate Visualizations
Massachusetts Institute of Technology（麻省理工学院）；Max Planck Institute for Biogeochemistry（马克斯·普朗克生物地球化学研究所）；University of British Columbia（不列颠哥伦比亚大学）；Peaceful Society, Science and Innovation Foundation（和平协会科学与创新基金会）；University of Granada（格拉纳达大学）；Scripps Institute of Oceanography（斯克里普斯海洋研究所）；US Forest Service（美国林务局）；University of Oxford（牛津大学）；Riot Games（拳头游戏）；Pasteur Labs（巴斯德实验室）；Surgical Data Science Collective（外科数据科学集体）
School of Information, University of California, Berkeley（加州大学伯克利分校信息学院）；IBM Chief Analytics Office（IBM首席分析办公室）
Climate change has caused reductions in river runoffs and aquifer recharge resulting in an increasingly unsustainable crop water demand from reduced freshwater availability. Achieving food security while deploying water in a sustainable manner will continue to be a major challenge necessitating careful monitoring and tracking of agricultural water usage. Historically, monitoring water usage has been a slow and expensive manual process with many imperfections and abuses. Ma-chine learning and remote sensing developments have increased the ability to automatically monitor irrigation patterns, bu…
University of Chicago（芝加哥大学）；Argonne National Laboratory（阿贡国家实验室）
Clouds play a critical role in the Earth's energy budget and their potential changes are one of the largest uncertainties in future climate projections. However, the use of satellite observations to understand cloud feedbacks in a warming climate has been hampered by the simplicity of existing cloud classification schemes, which are based on single-pixel cloud properties rather than utilizing spatial structures and textures. Recent advances in computer vision enable the grouping of different patterns of images without using human-predefined labels, providing a novel means of automated cloud cl…
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；Unit 31008 of the PLA（中国人民解放军31008部队）
Motivated by the development of deep convolution neural networks (DCNNs), tremendous progress has been gained in the field of aircraft detection. These DCNNs based detectors mainly belong to top-down approaches, which first enumerate massive potential locations of objects with the form of rectangular regions, and then identify whether they are objects or not. Compared with these top-down approaches, this paper shows that aircraft detection via bottom-up approach still performs competitively in the era of deep learning. We present a novel one-stage and anchor-free aircraft detection model in a …
School of Artificial Intelligence and School of Automation, Xi’an University of Posts and Telecommunications（人工智能学院和自动化学院，西安邮电大学）；Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（电气电子工程系，香港理工大学）
Detecting Brick Kiln Infrastructure at Scale: Graph, Foundation, and Remote Sensing Models for Satellite Imagery Data
Planetary Health Informatics (PHI) Lab, University of Oxford, Oxford, UK.（行星健康信息学实验室，牛津大学，英国）；Sustainable Environment (FUSE) Lab, University of Hong Kong, Hong Kong, China.（可持续环境（FUSE）实验室，香港大学，中国）；School of Computer and IT, Beaconhouse National University, Lahore, Pakistan（计算机与信息技术学院，贝肯豪斯国家大学，巴基斯坦）；Computer Science Department, Lahore University of Management Sciences, Lahore, Pakistan（管理科学大学计算机科学系，拉合尔，巴基斯坦）；Computer Science Department, University of Sarghoda, Sarghoda, Pakistan（计算机科学系，萨尔戈达大学，巴基斯坦）
University of Western São Paulo (UNOESTE)（西圣保罗大学（UNOESTE））；Federal University of Mato Grosso do Sul (UFMS)（南马托格罗索联邦大学（UFMS））；São Paulo State University (UNESP)（圣保罗州立大学（UNESP））；Brazilian Agricultural Research Agency（巴西农业研究公司）；University of Waterloo（滑铁卢大学）
University of St. Gallen（圣加仑大学）；Sociovestix Labs Ltd.（Sociovestix实验室有限公司）；ISS Inc.（ISS公司）
The major driver of global warming has been identified as the anthropogenic release of greenhouse gas (GHG) emissions from industrial activities. The quantitative monitoring of these emissions is mandatory to fully understand their effect on the Earth's climate and to enforce emission regulations on a large scale. In this work, we investigate the possibility to detect and quantify industrial smoke plumes from globally and freely available multi-band image data from ESA's Sentinel-2 satellites. Using a modified ResNet-50, we can detect smoke plumes of different sizes with an accuracy of 94.3%. …
King’s College London（伦敦国王学院）；UN Environment World Conservation Monitoring Centre（联合国环境规划署世界保护监测中心）
Dam reservoirs play an important role in meeting sustainable development goals and global climate targets. However, particularly for small dam reservoirs, there is a lack of consistent data on their geographical location. To address this data gap, a promising approach is to perform automated dam reservoir extraction based on globally available remote sensing imagery. It can be considered as a fine-grained task of water body extraction, which involves extracting water areas in images and then separating dam reservoirs from natural water bodies. We propose a novel deep neural network (DNN) based…
AB2CD: AI for Building Climate Damage Classification and Detection
IBM Consulting, Germany（IBM咨询公司（德国））；Karlsruhe Institute of Technology, Germany（卡尔斯鲁厄理工学院）；IBM Research – Europe, Switzerland（IBM欧洲研究院）；University of Bayreuth, Germany（拜罗伊特大学）
DeepAndes: A Self-Supervised Vision Foundation Model for Multi-Spectral Remote Sensing Imagery of the Andes
Department of Electrical and Computer Engineering, Vanderbilt University（电气与计算机工程系，范德比尔特大学）；Department of Anthropology, Vanderbilt University（人类学系，范德比尔特大学）；Data Science Institute, Vanderbilt University（数据科学研究院，范德比尔特大学）；Department of Computer Science, Vanderbilt University（计算机科学系，范德比尔特大学）；Department of Mathematics, Vanderbilt University（数学系，范德比尔特大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；Department of Anthropology, Brown University（人类学系，布朗大学）；Department of Radiology, Weill Cornell Medicine（放射学系，韦尔·科恩医学中心）
Technical University of Munich（慕尼黑工业大学）；Information Engineering University（信息工程大学）；Ruhr-University Bochum（波鸿鲁尔大学）；German Aerospace Center（德国航空航天中心）
As a unique classification scheme for urban forms and functions, the local climate zone (LCZ) system provides essential general information for any studies related to urban environments, especially on a large scale. Remote sensing data-based classification approaches are the key to large-scale mapping and monitoring of LCZs. The potential of deep learning-based approaches is not yet fully explored, even though advanced convolutional neural networks (CNNs) continue to push the frontiers for various computer vision tasks. One reason is that published studies are based on different datasets, usua…
FoMo: Multi-Modal, Multi-Scale and Multi-Task Remote Sensing Foundation Models for Forest Monitoring
National Observatory of Athens（雅典国家天文台）；National Technical University of Athens（雅典国家技术大学）；Harokopio University of Athens（哈罗科皮奥大学）；McGill University（麦吉尔大学）
Large Scale Masked Autoencoding for Reducing Label Requirements on SAR Data
University of Cambridge（剑桥大学）；Universidad Nacional de Colombia（哥伦比亚国立大学）；Universitat de València（瓦伦西亚大学）；University of Oxford（牛津大学）；Universidad de Antioquia（安蒂奥基亚大学）；European Space Agency（欧洲空间局）
University of Chicago（芝加哥大学）；Argonne National Laboratory（阿贡国家实验室）
Clouds play an important role in the Earth's energy budget and their behavior is one of the largest uncertainties in future climate projections. Satellite observations should help in understanding cloud responses, but decades and petabytes of multispectral cloud imagery have to date received only limited use. This study reduces the dimensionality of satellite cloud observations by grouping them via a novel automated, unsupervised cloud classification technique by using a convolutional neural network. Our technique combines a rotation-invariant autoencoder with hierarchical agglomerative cluste…
