We address the problem of cross-modal information retrieval in the domain of remote sensing. In particular, we are interested in two application scenarios: i) cross-modal retrieval between panchromatic (PAN) and multi-spectral imagery, and ii) multi-label image retrieval between very high resolution (VHR) images and speech based label annotations. Notice that these multi-modal retrieval scenarios are more challenging than the traditional uni-modal retrieval approaches given the inherent differences in distributions between the modalities. However, with the growing availability of multi-source remote sensing data and the scarcity of enough semantic annotations, the task of multi-modal retrieval has recently become extremely important. In this regard, we propose a novel deep neural network based architecture which is considered to learn a discriminative shared feature space for all the input modalities, suitable for semantically coherent information retrieval. Extensive experiments are carried out on the benchmark large-scale PAN - multi-spectral DSRSID dataset and the multi-label UC-Merced dataset. Together with the Merced dataset, we generate a corpus of speech signals corresponding to the labels. Superior performance with respect to the current state-of-the-art is observed in all the cases.
CR-JEPA: Cross-Modal Joint-Embedding Predictive Learning for Remote Sensing Image Retrieval
Space Applications Centre, Indian Space Research Organisation（印度空间研究组织空间应用中心）；Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay（印度理工学院孟买资源工程研究中心）
The development of cross-modal retrieval systems that can search and retrieve semantically relevant data across different modalities based on a query in any modality has attracted great attention in remote sensing (RS). In this paper, we focus our attention on cross-modal text-image retrieval, where queries from one modality (e.g., text) can be matched to archive entries from another (e.g., image). Most of the existing cross-modal text-image retrieval systems in RS require a high number of labeled training samples and also do not allow fast and memory-efficient retrieval. These issues limit th…
Due to the availability of multi-modal remote sensing (RS) image archives, one of the most important research topics is the development of cross-modal RS image retrieval (CM-RSIR) methods that search semantically similar images across different modalities. Existing CM-RSIR methods require the availability of a high quality and quantity of annotated training images. The collection of a sufficient number of reliable labeled images is time consuming, complex and costly in operational scenarios, and can significantly affect the final accuracy of CM-RSIR. In this paper, we introduce a novel self-su…
Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification
Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin（柏林工业大学电机工程与计算机科学学院）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）
A Multi-Modal Federated Learning Framework for Remote Sensing Image Classification
Technische Universität Berlin（柏林工业大学）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Due to the availability of large-scale multi-modal data (e.g., satellite images acquired by different sensors, text sentences, etc) archives, the development of cross-modal retrieval systems that can search and retrieve semantically relevant data across different modalities based on a query in any modality has attracted great attention in RS. In this paper, we focus our attention on cross-modal text-image retrieval, where queries from one modality (e.g., text) can be matched to archive entries from another (e.g., image). Most of the existing cross-modal text-image retrieval systems require a h…
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Remote sensing (RS) cross-modal text-image retrieval has attracted extensive attention for its advantages of flexible input and efficient query. However, traditional methods ignore the characteristics of multi-scale and redundant targets in RS image, leading to the degradation of retrieval accuracy. To cope with the problem of multi-scale scarcity and target redundancy in RS multimodal retrieval task, we come up with a novel asymmetric multimodal feature matching network (AMFMN). Our model adapts to multi-scale feature inputs, favors multi-source retrieval methods, and can dynamically filter r…
Technische Universität Berlin（柏林工业大学）；Direção-Geral do Território (DGT)（领土总局）；NOVA Information Management School (NOVA IMS), Universidade Nova Lisboa（里斯本新大学NOVA信息管理学院）
This paper presents the multi-modal BigEarthNet (BigEarthNet-MM) benchmark archive made up of 590,326 pairs of Sentinel-1 and Sentinel-2 image patches to support the deep learning (DL) studies in multi-modal multi-label remote sensing (RS) image retrieval and classification. Each pair of patches in BigEarthNet-MM is annotated with multi-labels provided by the CORINE Land Cover (CLC) map of 2018 based on its thematically most detailed Level-3 class nomenclature. Our initial research demonstrates that some CLC classes are challenging to be accurately described by only considering (single-date) B…
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Cross-modal remote sensing text-image retrieval (RSCTIR) has recently become an urgent research hotspot due to its ability of enabling fast and flexible information extraction on remote sensing (RS) images. However, current RSCTIR methods mainly focus on global features of RS images, which leads to the neglect of local features that reflect target relationships and saliency. In this article, we first propose a novel RSCTIR framework based on global and local information (GaLR), and design a multi-level information dynamic fusion (MIDF) module to efficaciously integrate features of different le…
Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing
Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin（柏林工业大学电机工程与计算机科学学院）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Environmental Computational Science and Earth Observation Laboratory (ECEO)（环境计算科学与地球观测实验室）
Zero-shot sketch-based remote sensing image retrieval based on multi-level and attention-guided tokenization
Anhui Province Key Laboratory of Wetland Ecosystem Protection and Restoration, Anhui University（安徽大学安徽省湿地生态保护与修复重点实验室）；School of Resources and Environmental Engineering, Anhui University（安徽大学资源与环境工程学院）；Shanghai Ubiquitous Navigation Technology Co. Ltd（上海普适导航科技股份有限公司）；The 54th Research Institute of China Electronics Technology Group Corporation（中国电子科技集团公司第五十四研究所）
Centre for Research-based Innovation Visual Intelligence（可视化研究型创新中心）
Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth's surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based o…
School of Aerospace Engineering, Tsinghua University（清华大学航天航空学院）
Cross-modality fusing complementary information of multispectral remote sensing image pairs can improve the perception ability of detection algorithms, making them more robust and reliable for a wider range of applications, such as nighttime detection. Compared with prior methods, we think different features should be processed specifically, the modality-specific features should be retained and enhanced, while the modality-shared features should be cherry-picked from the RGB and thermal IR modalities. Following this idea, a novel and lightweight multispectral feature fusion approach with joint…
Technical University of Munich（慕尼黑工业大学）；German Aerospace Center（德国航空航天中心）；Bundeswehr University Munich（慕尼黑联邦国防军大学）
About half of all optical observations collected via spaceborne satellites are affected by haze or clouds. Consequently, cloud coverage affects the remote sensing practitioner's capabilities of a continuous and seamless monitoring of our planet. This work addresses the challenge of optical satellite image reconstruction and cloud removal by proposing a novel multi-modal and multi-temporal data set called SEN12MS-CR-TS. We propose two models highlighting the benefits and use cases of SEN12MS-CR-TS: First, a multi-modal multi-temporal 3D-Convolution Neural Network that predicts a cloud-free imag…
DeepLight: Reconstructing High-Resolution Observations of Nighttime Light With Multi-Modal Remote Sensing Data
Tsinghua University - Xi’an Institute of Surveying and Mapping Joint Research Center（清华大学-西安测绘联合研究中心）；National supercomputing center in Shenzhen, Tsinghua University（清华大学深圳国家超级计算中心）；The University of Hong Kong（香港大学）；Sun-Yat San University（中山大学）
Multi-modal Uncertainty Robust Tree Cover Segmentation For High-Resolution Remote Sensing Images
School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）；National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing（空间智能信息处理国家重点实验室）；Department of Electrical and Computer Engineering, University of Delaware（电气与计算机工程系，德雷塞尔大学）；Swiss Federal Institute for Forest, Snow, and Landscape Research WSL（瑞士森林、雪和景观研究联邦 institute WSL）
Prototype-Based Information Compensation Network for Multi-Source Remote Sensing Data Classification
State Key Laboratory of Physical Oceanography, Ocean University of China（海洋大学物理海洋国家重点实验室）；Department of Electrical and Computer Engineering, Mississippi State University（密苏里州立大学电气与计算机工程系）
Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay（印度理工学院孟买分校资源工程研究中心）
As the field of remote sensing is evolving, we witness the accumulation of information from several modalities, such as multispectral (MS), hyperspectral (HSI), LiDAR etc. Each of these modalities possess its own distinct characteristics and when combined synergistically, perform very well in the recognition and classification tasks. However, fusing multiple modalities in remote sensing is cumbersome due to highly disparate domains. Furthermore, the existing methods do not facilitate cross-modal interactions. To this end, we propose a novel transformer based fusion method for HSI and LiDAR mod…
In recent years, due to the wide application of multi-sensor vision systems, multimodal image acquisition technology has continued to develop, and the registration problem based on multimodal images has gradually emerged. Most of the existing multimodal image registration methods are only suitable for two modalities, and cannot uniformly register multiple modal image data. Therefore, this paper proposes a multimodal remote sensing image registration method based on adaptive multi-scale PIIFD(AM-PIIFD). This method extracts KAZE features, which can effectively retain edge feature information wh…
Enhancing Scene Classification in Cloudy Image Scenarios: A Collaborative Transfer Method with Information Regulation Mechanism using Optical Cloud-Covered and SAR Remote Sensing Images
School of Geosciences and Info-Physics, Central South University（中南大学地球科学与信息物理学院）；Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente（特文特大学地球信息科学与地球观测学院）
Dynamic Frequency Feature Fusion Network for Multi-Source Remote Sensing Data Classification
State Key Laboratory of Physical Oceanography, Ocean University of China（中国海洋大学物理海洋学国家重点实验室）；Department of Electrical and Computer Engineering, Mississippi State University（密苏里州立大学电气与计算机工程系）
Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；GIPSA-lab（GIPSA实验室）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；the University of Tokyo（东京大学）；RIKEN Center for Advanced Intelligence Project (AIP), RIKEN（理化学研究所革新智能综合研究中心）；Xi’an Jiaotong University（西安交通大学）；INRIA（法国国家信息与自动化研究所）；LJK（让·昆兹曼实验室）；Mississippi State University（密西西比州立大学）
Classification and identification of the materials lying over or beneath the Earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS) and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the…
Archetypal scenarios for change detection generally consider two images acquired through sensors of the same modality. However, in some specific cases such as emergency situations, the only images available may be those acquired through sensors of different modalities. This paper addresses the problem of unsupervisedly detecting changes between two observed images acquired by sensors of different modalities with possibly different resolutions. These sensor dissimilarities introduce additional issues in the context of operational change detection that are not addressed by most of the classical …
COMIC: An Unsupervised Change Detection Method for Heterogeneous Remote Sensing Images Based on Copula Mixtures and Cycle-Consistent Adversarial Networks
In the past decades, absorption modalities have emerged as powerful tools for label-free functional and structural imaging of cells and tissues. Many biomolecules present unique absorption spectra providing chromophore-specific information on properties such as chemical bonding, and sample composition. As chromophores absorb photons the absorbed energy is emitted as photons (radiative relaxation) or converted to heat and under specific conditions pressure (non-radiative relaxation). Modalities like fluorescence microscopy may capture radiative relaxation to provide contrast, while modalities l…
