Learning a Joint Embedding of Multiple Satellite Sensors: A Case Study for Lake Ice Monitoring
Swiss Federal Institute of Technology, Zurich（苏黎世联邦理工学院）；Swiss Federal Institute of Aquatic Science and Technology（瑞士联邦水科学与技术研究所）；University of Zurich（苏黎世大学）
Manu Tom, Yuchang Jiang, Emmanuel Baltsavias, Konrad Schindler
发表信息：
IEEE Transactions on Geoscience and Remote Sensing 2022
Fusing satellite imagery acquired with different sensors has been a long-standing challenge of Earth observation, particularly across different modalities such as optical and Synthetic Aperture Radar (SAR) images. Here, we explore the joint analysis of imagery from different sensors in the light of representation learning: we propose to learn a joint embedding of multiple satellite sensors within a deep neural network. Our application problem is the monitoring of lake ice on Alpine lakes. To reach the temporal resolution requirement of the Swiss Global Climate Observing System (GCOS) office, we combine three image sources: Sentinel-1 SAR (S1-SAR), Terra MODIS, and Suomi-NPP VIIRS. The large gaps between the optical and SAR domains and between the sensor resolutions make this a challenging instance of the sensor fusion problem. Our approach can be classified as a late fusion that is learned in a data-driven manner. The proposed network architecture has separate encoding branches for each image sensor, which feed into a single latent embedding. I.e., a common feature representation shared by all inputs, such that subsequent processing steps deliver comparable output irrespective of which sort of input image was used. By fusing satellite data, we map lake ice at a temporal resolution of < 1.5 days. The network produces spatially explicit lake ice maps with pixel-wise accuracies > 91% (respectively, mIoU scores > 60%) and generalises well across different lakes and winters. Moreover, it sets a new state-of-the-art for determining the important ice-on and ice-off dates for the target lakes, in many cases meeting the GCOS requirement.
Lake ice, as part of the Essential Climate Variable (ECV) lakes, is an important indicator to monitor climate change and global warming. The spatio-temporal extent of lake ice cover, along with the timings of key phenological events such as freeze-up and break-up, provide important cues about the local and global climate. We present a lake ice monitoring system based on the automatic analysis of Sentinel-1 Synthetic Aperture Radar (SAR) data with a deep neural network. In previous studies that used optical satellite imagery for lake ice monitoring, frequent cloud cover was a main limiting fact…
TU Munich（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）；nFrames GmbH（nFrames有限公司）
Lake ice is a strong climate indicator and has been recognised as part of the Essential Climate Variables (ECV) by the Global Climate Observing System (GCOS). The dynamics of freezing and thawing, and possible shifts of freezing patterns over time, can help in understanding the local and global climate systems. One way to acquire the spatio-temporal information about lake ice formation, independent of clouds, is to analyse webcam images. This paper intends to move towards a universal model for monitoring lake ice with freely available webcam data. We demonstrate good performance, including the…
ETH Zürich (ETHZ)（苏黎世联邦理工学院）；University of Bern (UniBe)（伯尔尼大学）；Swiss Federal Institute of Aquatic Science and Technology (EAWAG)（瑞士水生科学与技术联邦研究所）
Various lake observables, including lake ice, are related to climate and climate change and provide a good opportunity for long-term monitoring. Lakes (and as part of them lake ice) is therefore considered an Essential Climate Variable (ECV) of the Global Climate Observing System (GCOS). Following the need for an integrated multi-temporal monitoring of lake ice in Switzerland, MeteoSwiss in the framework of GCOS Switzerland supported this 2-year project to explore not only the use of satellite images but also the possibilities of Webcams and in-situ measurements. The aim of this project is to …
Continuous observation of climate indicators, such as trends in lake freezing, is important to understand the dynamics of the local and global climate system. Consequently, lake ice has been included among the Essential Climate Variables (ECVs) of the Global Climate Observing System (GCOS), and there is a need to set up operational monitoring capabilities. Multi-temporal satellite images and publicly available webcam streams are among the viable data sources to monitor lake ice. In this work we investigate machine learning-based image analysis as a tool to determine the spatio-temporal extent …
ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）；Swiss Federal Institute of Aquatic Science and Technology（瑞士联邦水科学与技术研究所）
Depleting lake ice is a climate change indicator, just like sea-level rise or glacial retreat. Monitoring Lake Ice Phenology (LIP) is useful because long-term freezing and thawing patterns serve as sentinels to understand regional and global climate change. We report a study for the Oberengadin region of Switzerland, where several small- and medium-sized mountain lakes are located. We observe the LIP events, such as freeze-up, break-up and ice cover duration, across two decades (2000-2020) from optical satellite images. We analyse the time series of MODIS imagery by estimating spatially resolv…
Sea Ice Segmentation From SAR Data by Convolutional Transformer Networks
CEOSpaceTech, University Politehnica of Bucharest（布加勒斯特理工大学空间技术研究中心）；Remote Sensing Technology Institute, German Aerospace Center (DLR)（德国航空航天中心（DLR）遥感技术研究所）
University of Strasbourg（斯特拉斯堡大学）；Dedan Kimathi University of Technology（德丹·基马蒂理工大学）
Time-series of satellite images may reveal important data about changes in environmental conditions and natural or urban landscape structures that are of potential interest to citizens, historians, or policymakers. We applied a fast method of image analysis using Self Organized Maps (SOM) and, more specifically, the quantization error (QE), for the visualization of critical changes in satellite images of Las Vegas, generated across the years 1984-2008, a period of major restructuration of the urban landscape. As shown in our previous work, the QE from the SOM output is a reliable measure of va…
RoughNet: Mapping Arctic Sea Ice Roughness Using Diffusion-Based Super-Resolution of Satellite Imagery
University College London（伦敦大学学院）；Alfred Wegener Institute Helmholtz Centre for Polar and Marine Research（阿尔弗雷德·韦格纳极地与海洋研究所亥姆霍兹中心）；University of Victoria（维多利亚大学）
From Remote Sensing to Multiple Time Horizons Forecasts: Transformers Model for CyanoHAB Intensity in Lake Champlain
Vermont Artificial Intelligence Laboratory (VaiL)（佛蒙特人工智能实验室（VaiL））；Department of Computer Science, University of Vermont（大学计算机科学系）；University of Vermont（佛蒙特大学）；Department of Community Development and Applied Economics & Department of Computer Science, University of Vermont（社区发展与应用经济学系及计算机科学系，佛蒙特大学）；Department of Geology, University of Vermont（地质学系，佛蒙特大学）；Department of Civil and Environmental Engineering, University of Vermont（土木与环境工程系，佛蒙特大学）；Vermont EPSCoR (Established Program to Stimulate Competitive Research), University of Vermont（佛蒙特EPSCoR（促进竞争性研究计划），佛蒙特大学）；Vermont Department of Environmental Conservation（佛蒙特州环境保护部）；Water Quality Solutions, Inc.（水质解决方案公司）
Data-Driven Uncertainty-Aware Forecasting of Sea Ice Conditions in the Gulf of Ob Based on Satellite Radar Imagery
Applied AI Center, Skolkovo Institute of Science and Technology, Moscow, Russia（斯克尔科沃科学与技术研究所应用人工智能中心）；Moscow Institute of Physics and Technology, Dolgoprudny, Russia（莫斯科物理技术学院）；Autonomous Non-Profit Organization Artificial Intelligence Research Institute (AIRI), Moscow, Russia（自主非营利组织人工智能研究 institute（AIRI））
IceCloudNet: 3D reconstruction of cloud ice from Meteosat SEVIRI
Institute for Atmospheric and Climate Science, ETH Zurich（苏黎世联邦理工学院大气与气候科学研究所）；European Space Agency (ESA)（欧洲空间局）；Consiglio Nazionale delle Ricerche - Istituto di Scienze Marine (CNR-ISMAR)（国家研究委员会海洋科学研究所）
IceCloudNet: Cirrus and mixed-phase cloud prediction from SEVIRI input learned from sparse supervision
ETH Zurich（苏黎世联邦理工学院）；European Space Agency (ESA)（欧洲空间局）；Consiglio Nazionale delle Ricerche - Istituto di Scienze Marine (CNR-ISMAR)（意大利国家研究委员会-海洋科学研究所）
During summer, melt ponds have a significant influence on Arctic sea-ice albedo. The melt pond fraction (MPF) also has the ability to forecast the Arctic sea-ice in a certain period. It is important to retrieve accurate melt pond fraction (MPF) from satellite data for Arctic research. This paper proposes a satellite MPF retrieval model based on the multi-layer neural network, named MPF-NN. Our model uses multi-spectral satellite data as model input and MPF information from multi-site and multi-period visible imagery as prior knowledge for modeling. It can effectively model melt ponds evolution…
University of Chicago（芝加哥大学）；IBM T.J. Watson Research Center（IBM T.J. 沃森研究中心）
We consider the problem of removing and replacing clouds in satellite image sequences, which has a wide range of applications in remote sensing. Our approach first detects and removes the cloud-contaminated part of the image sequences. It then recovers the missing scenes from the clean parts using the proposed "TECROMAC" (TEmporally Contiguous RObust MAtrix Completion) objective. The objective function balances temporal smoothness with a low rank solution while staying close to the original observations. The matrix whose the rows are pixels and columnsare days corresponding to the image, has l…
Global surface water detection in very-high-resolution (VHR) satellite imagery can directly serve major applications such as refined flood mapping and water resource assessment. Although achievements have been made in detecting surface water in small-size satellite images corresponding to local geographic scales, datasets and methods suitable for mapping and analyzing global surface water have yet to be explored. To encourage the development of this task and facilitate the implementation of relevant applications, we propose the GLH-water dataset that consists of 250 satellite images and manual…
U.S. Army Corps of Engineers, Cold Regions Research and Engineering Laboratory（美国陆军工程兵团寒区研究与工程实验室）
As Arctic conditions rapidly change, human activity in the Arctic will continue to increase and so will the need for high-resolution observations of sea ice. While satellite imagery can provide high spatial resolution, it is temporally sparse and significant ice deformation can occur between observations. This makes it difficult to apply feature tracking or image correlation techniques that require persistent features to exist between images. With this in mind, we propose a technique based on optimal transport, which is commonly used to measure differences between probability distributions. Wh…
