Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection
Department of Computer Engineering, Ankara University（安卡拉大学计算机工程系）；Department of Modeling and Simulation, Graduate School of Informatics, METU（信息学院建模与模拟系，METU）
Multimodal remote sensing data provide complementary information for semantic segmentation, but in real-world deployments, some modalities may be unavailable due to sensor failures, acquisition issues, or challenging atmospheric conditions. Existing multimodal segmentation models typically address missing modalities by learning a shared representation across inputs. However, this approach can introduce a trade-off by compromising modality-specific complementary information and reducing performance when all modalities are available. In this paper, we tackle this limitation with CBC-SLP, a multimodal semantic segmentation model designed to preserve both modality-invariant and modality-specific information. Inspired by the theoretical results on modality alignment, which state that perfectly aligned multimodal representations can lead to sub-optimal performance in downstream prediction tasks, we propose a novel structured latent projection approach as an architectural inductive bias. Rather than enforcing this strategy through a loss term, we incorporate it directly into the architecture. In particular, to use the complementary information effectively while maintaining robustness under random modality dropout, we structure the latent representations into shared and modality-specific components and adaptively transfer them to the decoder according to the random modality availability mask. Extensive experiments on three multimodal remote sensing image sets demonstrate that CBC-SLP consistently outperforms state-of-the-art multimodal models across full and missing modality scenarios. Besides, we empirically demonstrate that the proposed strategy can recover the complementary information that may not be preserved in a shared representation. The code is available at https://github.com/iremulku/Multispectral-Semantic-Segmentation-via-Structured-Latent-Projection-CBC-SLP-.
Unmanned aircraft have decreased the cost required to collect remote sensing imagery, which has enabled researchers to collect high-spatial resolution data from multiple sensor modalities more frequently and easily. The increase in data will push the need for semantic segmentation frameworks that are able to classify non-RGB imagery, but this type of algorithmic development requires an increase in publicly available benchmark datasets with class labels. In this paper, we introduce a high-resolution multispectral dataset with image labels. This new benchmark dataset has been pre-split into trai…
School of Aerospace Engineering, Tsinghua University（清华大学航天航空学院）
Cross-modality fusing complementary information of multispectral remote sensing image pairs can improve the perception ability of detection algorithms, making them more robust and reliable for a wider range of applications, such as nighttime detection. Compared with prior methods, we think different features should be processed specifically, the modality-specific features should be retained and enhanced, while the modality-shared features should be cherry-picked from the RGB and thermal IR modalities. Following this idea, a novel and lightweight multispectral feature fusion approach with joint…
From Words to Wavelengths: VLMs for Few-Shot Multispectral Object Detection
Univ Bretagne Sud, IRISA, UMR 6074（布列塔尼大学，IRISA，UMR 6074）；Univ Rennes, IRISA, UMR 6074（里尔大学，IRISA，UMR 6074）；ATERMES；UiT The Arctic University of Norway（北欧大学）
Optimizing Multispectral Object Detection: A Bag of Tricks and Comprehensive Benchmarks
Beijing Forestry University（北京林业大学）；Hong Kong University of Science and Technology（香港科技大学）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)（多模态人工智能系统国家重点实验室）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；Sun Yat-sen University（中山大学）
Chester F. Carlson Center for Imaging Science Rochester Institute of Technology（切斯特·F·卡尔森成像科学中心 罗切斯特理工学院）
Deep convolutional neural networks (DCNNs) have been used to achieve state-of-the-art performance on many computer vision tasks (e.g., object recognition, object detection, semantic segmentation) thanks to a large repository of annotated image data. Large labeled datasets for other sensor modalities, e.g., multispectral imagery (MSI), are not available due to the large cost and manpower required. In this paper, we adapt state-of-the-art DCNN frameworks in computer vision for semantic segmentation for MSI imagery. To overcome label scarcity for MSI data, we substitute real MSI for generated syn…
Multispectral image pairs can provide the combined information, making object detection applications more reliable and robust in the open world. To fully exploit the different modalities, we present a simple yet effective cross-modality feature fusion approach, named Cross-Modality Fusion Transformer (CFT) in this paper. Unlike prior CNNs-based works, guided by the transformer scheme, our network learns long-range dependencies and integrates global contextual information in the feature extraction stage. More importantly, by leveraging the self attention of the transformer, the network can natu…
MO R-CNN: Multispectral Oriented R-CNN for Object Detection in Remote Sensing Image
College of Electronics and Information Engineering, Sichuan University（四川大学电子信息工程学院）；School of Mechanical Engineering and Automation, Fuzhou University（福州大学机械工程与自动化学院）；Yunnan Key Laboratory of Software Engineering, Yunnan University（云南软件工程重点实验室）
针对多光谱定向目标检测中模态差异与计算开销问题，提出轻量级框架 MO R-CNN，通过异构特征提取、单模态监督和条件标签融合提升检测性能。
DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensing
University of Electronic Science and Technology of China（电子科技大学）；Beijing Institute of Space Mechanics and Electricity（北京空间机电研究所）；Donghai Laboratory（东海实验室）
YOLOv11-RGBT: Towards a Comprehensive Single-Stage Multispectral Object Detection Framework
School of Instrument Science and Opto-electronics Engineering, Anhui Province Key Laboratory of Measuring Theory and Precision Instrument（仪器科学与光电工程学院、安徽省测量理论与精密仪器重点实验室）；Hefei University of Technology（合肥工业大学）；School of Information Engineering（信息工程学院）；Engineering University of PAP（工程大学）
Multispectral Detection Transformer with Infrared-Centric Feature Fusion
Artificial Intelligence Graduate School, Gwangju Institute of Science and Technology (GIST)（人工智能研究生院，全州科学技术院（GIST））；School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology (GIST)（电气工程与计算机科学学院，全州科学技术院（GIST））
DPDETR: Decoupled Position Detection Transformer for Infrared-Visible Object Detection
School of Communications and Information Engineering, Chongqing University of Posts and Telecommunications（通信与信息工程学院，重庆邮电大学）；School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（智能系统工程学院，中山大学深圳校区）；School of Mathematics and Statistics, Xi’an Jiaotong University（数学与统计学院，西安交通大学）
Multispectral pedestrian detection is capable of adapting to insufficient illumination conditions by leveraging color-thermal modalities. On the other hand, it is still lacking of in-depth insights on how to fuse the two modalities effectively. Compared with traditional pedestrian detection, we find multispectral pedestrian detection suffers from modality imbalance problems which will hinder the optimization process of dual-modality network and depress the performance of detector. Inspired by this observation, we propose Modality Balance Network (MBNet) which facilitates the optimization proce…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Renmin University of China（中国人民大学）；CEBSIT, Chinese Academy of Sciences（中国科学院CEBSIT）
Multispectral pedestrian detection has shown great advantages under poor illumination conditions, since the thermal modality provides complementary information for the color image. However, real multispectral data suffers from the position shift problem, i.e. the color-thermal image pairs are not strictly aligned, making one object has different positions in different modalities. In deep learning based methods, this problem makes it difficult to fuse the feature maps from both modalities and puzzles the CNN training. In this paper, we propose a novel Aligned Region CNN (AR-CNN) to handle the w…
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；University of Iceland（冰岛大学）；RIKEN（日本理化学研究所）
Due to the ever-growing diversity of the data source, multi-modality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multi-modalities that exist in both training and test sets, yet they are less investigated in absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multi-modalities in the training process. By this way, the out-of-sample from any of multi-modalities can be directly projected onto the learned space for a more e…
MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Research Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）
Multimodal emotion recognition leverages complementary information across modalities to gain performance. However, we cannot guarantee that the data of all modalities are always present in practice. In the studies to predict the missing data across modalities, the inherent difference between heterogeneous modalities, namely the modality gap, presents a challenge. To address this, we propose to use invariant features for a missing modality imagination network (IF-MMIN) which includes two novel mechanisms: 1) an invariant feature learning strategy that is based on the central moment discrepancy …
Shanghai Jiao Tong University（上海交通大学）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（系统控制与信息处理教育部重点实验室）；Science and Technology on Near-Surface Detection Laboratory（近地面探测技术重点实验室）；University of Michigan-Shanghai Jiao Tong University Joint Institute（密西根大学-上海交通大学联合学院）
Thermal infrared (TIR) image has proven effectiveness in providing temperature cues to the RGB features for multispectral pedestrian detection. Most existing methods directly inject the TIR modality into the RGB-based framework or simply ensemble the results of two modalities. This, however, could lead to inferior detection performance, as the RGB and TIR features generally have modality-specific noise, which might worsen the features along with the propagation of the network. Therefore, this work proposes an effective and efficient cross-modality fusion module called Bi-directional Adaptive A…
Multispectral images of color-thermal pairs have shown more effective than a single color channel for pedestrian detection, especially under challenging illumination conditions. However, there is still a lack of studies on how to fuse the two modalities effectively. In this paper, we deeply compare six different convolutional network fusion architectures and analyse their adaptations, enabling a vanilla architecture to obtain detection performances comparable to the state-of-the-art results. Further, we discover that pedestrian detection confidences from color or thermal images are correlated …
Multispectral pedestrian detection has attracted increasing attention from the research community due to its crucial competence for many around-the-clock applications (e.g., video surveillance and autonomous driving), especially under insufficient illumination conditions. We create a human baseline over the KAIST dataset and reveal that there is still a large gap between current top detectors and human performance. To narrow this gap, we propose a network fusion architecture, which consists of a multispectral proposal network to generate pedestrian proposals, and a subsequent multispectral cla…
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）；RIKEN（理化学研究所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；University of Iceland（冰岛大学）
With a large amount of open satellite multispectral imagery (e.g., Sentinel-2 and Landsat-8), considerable attention has been paid to global multispectral land cover classification. However, its limited spectral information hinders further improving the classification performance. Hyperspectral imaging enables discrimination between spectrally similar classes but its swath width from space is narrow compared to multispectral ones. To achieve accurate land cover classification over a large coverage, we propose a cross-modality feature learning framework, called common subspace learning (CoSpace…
CCSD: Cross-Modal Compositional Self-Distillation for Robust Brain Tumor Segmentation with Missing Modalities
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（上海智能自主系统研究院，同济大学）；College of Biomedical Engineering, Fudan University（生物医学工程学院，复旦大学）；School of Medicine, Tongji University（医学院，同济大学）；College of Information Engineering, China Jiliang University（信息工程学院，中国嘉兴大学）
Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification
GFZ Helmholtz Center for Geosciences（德国波茨坦地学研究中心亥姆霍兹中心）；INRAE, UMR TETIS, University of Montpellier（法国蒙彼利埃大学农业环境与生态研究院、蒙彼利埃大学TETIS联合研究单位）；INRIA, EVERGREEN, University of Montpellier（法国蒙彼利埃大学信息与自动化研究所、EVERGREEN实验室）；CIRAD, UMR TETIS, University of Montpellier（法国蒙彼利埃大学国际农业研究磋商组织、蒙彼利埃大学TETIS联合研究单位）
Remote sensing data imputation using deep learning for multispectral imagery
Water Research Centre, University of New South Wales（新南威尔士大学水研究中心）；ARC ITTC Data Analytics for Resources and Environments, University of New South Wales（新南威尔士大学资源与环境数据分析师联盟）；Transitional Artificial Intelligence Research Group, School of Mathematics and Statistics, University of New South Wales（新南威尔士大学数学与统计学过渡人工智能研究组）
TU Kaiserslautern（凯泽斯劳滕工业大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
The integration of information acquired with different modalities, spatial resolution and spectral bands has shown to improve predictive accuracies. Data fusion is therefore one of the key challenges in remote sensing. Most prior work focusing on multi-modal fusion, assumes that modalities are always available during inference. This assumption limits the applications of multi-modal models since in practice the data collection process is likely to generate data with missing, incomplete or corrupted modalities. In this paper, we show that Generative Adversarial Networks can be effectively used t…
Beijing-Dublin International College（北京-都柏林国际学院）；University College Dublin（都柏林大学）
This technical report presents a comparative analysis of existing deep learning (DL) based approaches for brain tumor segmentation with missing MRI modalities. Approaches evaluated include the Adversarial Co-training Network (ACN) and a combination of mmGAN and DeepMedic. A more stable and easy-to-use version of mmGAN is also open-sourced at a GitHub repository. Using the BraTS2018 dataset, this work demonstrates that the state-of-the-art ACN performs better especially when T1c is missing. While a simple combination of mmGAN and DeepMedic also shows strong potentials when only one MRI modality…
Zhejiang University-University of Illinois at Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳香槟分校联合学院）；Zhejiang University（浙江大学）
Multimodal sentiment analysis (MSA) is an important way of observing mental activities with the help of data captured from multiple modalities. However, due to the recording or transmission error, some modalities may include incomplete data. Most existing works that address missing modalities usually assume a particular modality is completely missing and seldom consider a mixture of missing across multiple modalities. In this paper, we propose a simple yet effective meta-sampling approach for multimodal sentiment analysis with missing modalities, namely Missing Modality-based Meta Sampling (M3…
Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Engineering and Technology Taxila（塔克西拉工程大学）；Dalhousie University（达尔豪斯大学）；Interuniversity Microelectronics Centre (IMEC)（校际微电子中心）；Johannes Kepler University Linz（约翰开普勒林茨大学）；Linz Institute of Technology（林茨理工学院）
Elekta（医科达）；KTH - Royal institute of Technology（KTH皇家理工学院）
Over the last few years machine learning has demonstrated groundbreaking results in many areas of medical image analysis, including segmentation. A key assumption, however, is that the train- and test distributions match. We study a realistic scenario where this assumption is clearly violated, namely segmentation with missing input modalities. We describe two neural network approaches that can handle a variable number of input modalities. The first is modality dropout: a simple but surprisingly effective modification of the training. The second is the unified representation network: a network …
UiT The Arctic University of Norway（挪威北极大学）；Norwegian Computing Center（挪威计算中心）
Automatic urban land cover classification is a fundamental problem in remote sensing, e.g. for environmental monitoring. The problem is highly challenging, as classes generally have high inter-class and low intra-class variance. Techniques to improve urban land cover classification performance in remote sensing include fusion of data from different sensors with different data modalities. However, such techniques require all modalities to be available to the classifier in the decision-making process, i.e. at test time, as well as in training. If a data modality is missing at test time, current …
ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
NepAl Applied Mathematics and Informatics Institute（尼泊尔应用数学与信息学研究所）；Fogsphere (Redev.AI Ltd)（福格球（Redev.AI有限公司））；University of Lausanne（洛桑大学）；West Virginia University（西弗吉尼亚大学）；University of Verona（维罗纳大学）；University College London（伦敦大学学院）；University of Aberdeen（阿伯丁大学）
Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization
Institute for Clarity in Documentation（清晰文档研究所）；Inria Paris-Rocquencourt（巴黎-鲁克桑克鲁特研究所）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）；Shanghai Jiao Tong University（上海交通大学）；Hohai University（河海大学）；Shaanghai Jiao Tong University（上海交通大学）
University at Buffalo, The State University of New York（纽约州立大学布法罗分校）
Brain Tumor Segmentation from magnetic resonance imaging (MRI) is a critical technique for early diagnosis. However, rather than having complete four modalities as in BraTS dataset, it is common to have missing modalities in clinical scenarios. We design a brain tumor segmentation algorithm that is robust to the absence of any modality. Our network includes a channel-independent encoding path and a feature-fusion decoding path. We use self-supervised training through channel dropout and also propose a novel domain adaptation method on feature maps to recover the information from the missing ch…
St. Pölten University of Applied Sciences（圣珀尔滕应用科学大学）；Kufstein University of Applied Sciences（库夫施泰因应用科学大学）
The assessment and valuation of real estate requires large datasets with real estate information. Unfortunately, real estate databases are usually sparse in practice, i.e., not for each property every important attribute is available. In this paper, we study the potential of predicting high-level real estate attributes from visual data, specifically from two visual modalities, namely indoor (interior) and outdoor (facade) photos. We design three models using different multimodal fusion strategies and evaluate them for three different use cases. Thereby, a particular challenge is to handle miss…
