Aerial object detection faces significant challenges in real-world scenarios, such as small objects and extensive background interference, which limit the performance of RGB-based detectors with insufficient discriminative information. Multispectral images (MSIs) capture additional spectral cues across multiple bands, offering a promising alternative. However, the lack of training data has been the primary bottleneck to exploiting the potential of MSIs. To address this gap, we introduce the first large-scale dataset for Multispectral Object Detection in Aerial images (MODA), which comprises 14,041 MSIs and 330,191 annotations across diverse, challenging scenarios, providing a comprehensive data foundation for this field. Furthermore, to overcome challenges inherent to aerial object detection using MSIs, we propose OSSDet, a framework that integrates spectral and spatial information with object-aware cues. OSSDet employs a cascaded spectral-spatial modulation structure to optimize target perception, aggregates spectrally related features by exploiting spectral similarities to reinforce intra-object correlations, and suppresses irrelevant background via object-aware masking. Moreover, cross-spectral attention further refines object-related representations under explicit object-aware guidance. Extensive experiments demonstrate that OSSDet outperforms existing methods with comparable parameters and efficiency.
SFFR: Spatial-Frequency Feature Reconstruction for Multispectral Aerial Object Detection
School of Computer, Jiangsu University of Science and Technology（江苏科技大学计算机学院）；School of Electrical and Information Engineering, Jiangsu University（江苏大学电气与信息工程学院）；School of Automation, Southeast University（东南大学自动化学院）
DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensing
University of Electronic Science and Technology of China（电子科技大学）；Beijing Institute of Space Mechanics and Electricity（北京空间机电研究所）；Donghai Laboratory（东海实验室）
Optimizing Multispectral Object Detection: A Bag of Tricks and Comprehensive Benchmarks
Beijing Forestry University（北京林业大学）；Hong Kong University of Science and Technology（香港科技大学）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)（多模态人工智能系统国家重点实验室）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；Sun Yat-sen University（中山大学）
Multispectral image pairs can provide the combined information, making object detection applications more reliable and robust in the open world. To fully exploit the different modalities, we present a simple yet effective cross-modality feature fusion approach, named Cross-Modality Fusion Transformer (CFT) in this paper. Unlike prior CNNs-based works, guided by the transformer scheme, our network learns long-range dependencies and integrates global contextual information in the feature extraction stage. More importantly, by leveraging the self attention of the transformer, the network can natu…
Evaluation of Deep Learning Semantic Segmentation for Land Cover Mapping on Multispectral, Hyperspectral and High Spatial Aerial Imagery
Center for Space and Remote Sensing Research, National Central University（国立中央大学太空及遥测研究中心）；Department of Computer Science and Information Engineering, National Central University（国立中央大学资讯工程学系）
From Words to Wavelengths: VLMs for Few-Shot Multispectral Object Detection
Univ Bretagne Sud, IRISA, UMR 6074（布列塔尼大学，IRISA，UMR 6074）；Univ Rennes, IRISA, UMR 6074（里尔大学，IRISA，UMR 6074）；ATERMES；UiT The Arctic University of Norway（北欧大学）
School of Aerospace Engineering, Tsinghua University（清华大学航天航空学院）
Cross-modality fusing complementary information of multispectral remote sensing image pairs can improve the perception ability of detection algorithms, making them more robust and reliable for a wider range of applications, such as nighttime detection. Compared with prior methods, we think different features should be processed specifically, the modality-specific features should be retained and enhanced, while the modality-shared features should be cherry-picked from the RGB and thermal IR modalities. Following this idea, a novel and lightweight multispectral feature fusion approach with joint…
Multispectral Detection Transformer with Infrared-Centric Feature Fusion
Artificial Intelligence Graduate School, Gwangju Institute of Science and Technology (GIST)（人工智能研究生院，全州科学技术院（GIST））；School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology (GIST)（电气工程与计算机科学学院，全州科学技术院（GIST））
Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detection
School of Electrical and Information Engineering, Jiangsu University, Zhenjiang, 212013, China（江苏大学电气与信息工程学院）；Department of Computer Science and Engineering, University of North Texas, Denton, TX 76207, USA（德克萨斯大学北卡罗来纳分校计算机科学与工程系）；School of Computer Science and Engineering, Jiangsu University of Science and Technology, Zhenjiang, 212003, China（江苏科技大学计算机科学与工程学院）；School of Automation, Southeast University, Nanjing, 210096, China（东南大学自动化学院）；Bodhi Intelligence Lab, Department of Artificial Intelligence, Westlake University, Hangzhou, Zhejiang 310030, China（西湖大学人工智能研究院）
YOLOv11-RGBT: Towards a Comprehensive Single-Stage Multispectral Object Detection Framework
School of Instrument Science and Opto-electronics Engineering, Anhui Province Key Laboratory of Measuring Theory and Precision Instrument（仪器科学与光电工程学院、安徽省测量理论与精密仪器重点实验室）；Hefei University of Technology（合肥工业大学）；School of Information Engineering（信息工程学院）；Engineering University of PAP（工程大学）
DAMSDet: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion
School of Communications and Information Engineering, Chongqing University of Posts and Telecommunications（重庆邮电大学通信与信息工程学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能系统工程学院）
MO R-CNN: Multispectral Oriented R-CNN for Object Detection in Remote Sensing Image
College of Electronics and Information Engineering, Sichuan University（四川大学电子信息工程学院）；School of Mechanical Engineering and Automation, Fuzhou University（福州大学机械工程与自动化学院）；Yunnan Key Laboratory of Software Engineering, Yunnan University（云南软件工程重点实验室）
针对多光谱定向目标检测中模态差异与计算开销问题，提出轻量级框架 MO R-CNN，通过异构特征提取、单模态监督和条件标签融合提升检测性能。
The Copernicus Sentinel-2 program now provides multispectral images at a global scale with a high revisit rate. In this paper we explore the usage of convolutional neural networks for urban change detection using such multispectral images. We first present the new change detection dataset that was used for training the proposed networks, which will be openly available to serve as a benchmark. The Onera Satellite Change Detection (OSCD) dataset is composed of pairs of multispectral aerial images, and the changes were manually annotated at pixel level. We then propose two architectures to detect…
School of Electrical and Information Engineering, Jiangsu University, Zhenjiang, 212013, China（江苏大学电气与信息工程学院）；School of Computer Science and Engineering, Jiangsu University of Science and Technology, Zhenjiang, 212003, China（江苏科技大学计算机科学与工程学院）；School of Automation, Southeast University, Nanjing, 210096, China（东南大学自动化学院）；Department of Computer Science and Engineering, University of North Texas, Denton, TX 76207, USA（北卡罗来纳州立大学计算机科学与工程系）；School of Computing, Nanjing Normal University, Nanjing, 210046, Jiangsu（南京师范大学计算机学院）
Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection
Department of Computer Engineering, Ankara University（安卡拉大学计算机工程系）；Department of Modeling and Simulation, Graduate School of Informatics, METU（信息学院建模与模拟系，METU）
Bridging the RGB-IR Gap: Consensus and Discrepancy Modeling for Text-Guided Multispectral Detection
Department of Automation, Tsinghua University（清华大学自动化系）；State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全全国重点实验室）；Tsinghua University（清华大学）
Integrating multispectral data in object detection, especially visible and infrared images, has received great attention in recent years. Since visible (RGB) and infrared (IR) images can provide complementary information to handle light variations, the paired images are used in many fields, such as multispectral pedestrian detection, RGB-IR crowd counting and RGB-IR salient object detection. Compared with natural RGB-IR images, we find detection in aerial RGB-IR images suffers from cross-modal weakly misalignment problems, which are manifested in the position, size and angle deviations of the …
Progressive Pixel-Neighborhood Deformable Cross-Attention for Multispectral Object Detection
School of Electrical and Information Engineering, Jiangsu University（江苏大学电气与信息工程学院）；School of Computer Science and Engineering, Jiangsu University of Science and Technology（江苏科技大学计算机科学与工程学院）
Object detection and classification using aerial images is a challenging task as the information regarding targets are not abundant. Synthetic Aperture Radar(SAR) images can be used for Automatic Target Recognition(ATR) systems as it can operate in all-weather conditions and in low light settings. But, SAR images contain salt and pepper noise(speckle noise) that cause hindrance for the deep learning models to extract meaningful features. Using just aerial view Electro-optical(EO) images for ATR systems may also not result in high accuracy as these images are of low resolution and also do not p…
Dual Sparse Aggregation Transformer for Multispectral Object Detection
Northwestern Polytechnical University（西北工业大学）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；School of Cybersecurity, Northwestern Polytechnical University（西北工业大学网络空间安全学院）；Shaanxi Provincial Key Laboratory of Speech and Image Information Processing（陕西省语音与图像信息处理重点实验室）；National Engineering Laboratory for Integrated Aerospace-Ground-Ocean Big Data Application Technology（空天地海一体化大数据应用技术国家工程实验室）
Unmanned aircraft have decreased the cost required to collect remote sensing imagery, which has enabled researchers to collect high-spatial resolution data from multiple sensor modalities more frequently and easily. The increase in data will push the need for semantic segmentation frameworks that are able to classify non-RGB imagery, but this type of algorithmic development requires an increase in publicly available benchmark datasets with class labels. In this paper, we introduce a high-resolution multispectral dataset with image labels. This new benchmark dataset has been pre-split into trai…
FoCal: Frequency-Oriented Cross-Modal Interaction and Spectral Calibration for Aerial Visible-Infrared Object Detection
Nanjing University of Science and Technology（南京理工大学）；Shanghai Institute of Technical Physics, Chinese Academy of Sciences（中国科学院上海技术物理研究所）；China Electronics Technology Group Corporation（中国电子科技集团公司）
UEMM-Air: Make Unmanned Aerial Vehicles Perform More Multi-modal Tasks
College of Computer Science and Software Engineering, Hohai University（河海大学计算机与软件学院）；College of Artificial Intelligence and Automation, Hohai University（河海大学人工智能与自动化学院）；Department of Computer Science and Engineering, Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）
DEPFusion: Dual-Domain Enhancement and Priority-Guided Mamba Fusion for UAV Multispectral Object Detection
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；School of Electronic and Information Engineering, South China University of Technology（华南理工大学电子信息学院）；School of Artificial Intelligence, XiDian University（西安电子科技大学人工智能学院）
University of Coimbra（科英布拉大学）；Institute for Systems Engineering and Computers at Coimbra（科英布拉系统工程与计算机研究所）；Polytechnic Institute of Coimbra（科英布拉理工学院）；Stockholm University（斯德哥尔摩大学）；Bolin Centre for Climate Research（博林气候研究中心）；Navarino Environmental Observatory（纳瓦里诺环境观测站）
Digital agriculture has evolved significantly over the last few years due to the technological developments in automation and computational intelligence applied to the agricultural sector, including vineyards which are a relevant crop in the Mediterranean region. In this work, a study is presented of semantic segmentation for vine detection in real-world vineyards by exploring state-of-the-art deep segmentation networks and conventional unsupervised methods. Camera data have been collected on vineyards using an Unmanned Aerial System (UAS) equipped with a dual imaging sensor payload, namely a …
Unmanned aerial vehicles (UAVs) equipped with multiple complementary sensors have tremendous potential for fast autonomous or remote-controlled semantic scene analysis, e.g., for disaster examination. In this work, we propose a UAV system for real-time semantic inference and fusion of multiple sensor modalities. Semantic segmentation of LiDAR scans and RGB images, as well as object detection on RGB and thermal images, run online onboard the UAV computer using lightweight CNN architectures and embedded inference accelerators. We follow a late fusion approach where semantic information from mult…
Unmanned aerial vehicles (UAVs) with mounted cameras have the advantage of capturing aerial (bird-view) images. The availability of aerial visual data and the recent advances in object detection algorithms led the computer vision community to focus on object detection tasks on aerial images. As a result of this, several aerial datasets have been introduced, including visual data with object annotations. UAVs are used solely as flying-cameras in these datasets, discarding different data types regarding the flight (e.g., time, location, internal sensors). In this work, we propose a multi-purpose…
Chester F. Carlson Center for Imaging Science Rochester Institute of Technology（切斯特·F·卡尔森成像科学中心 罗切斯特理工学院）
Deep convolutional neural networks (DCNNs) have been used to achieve state-of-the-art performance on many computer vision tasks (e.g., object recognition, object detection, semantic segmentation) thanks to a large repository of annotated image data. Large labeled datasets for other sensor modalities, e.g., multispectral imagery (MSI), are not available due to the large cost and manpower required. In this paper, we adapt state-of-the-art DCNN frameworks in computer vision for semantic segmentation for MSI imagery. To overcome label scarcity for MSI data, we substitute real MSI for generated syn…
Unmanned aerial vehicles (UAVs) equipped with multiple complementary sensors have tremendous potential for fast autonomous or remote-controlled semantic scene analysis, e.g., for disaster examination. Here, we propose a UAV system for real-time semantic inference and fusion of multiple sensor modalities. Semantic segmentation of LiDAR scans and RGB images, as well as object detection on RGB and thermal images, run online onboard the UAV computer using lightweight CNN architectures and embedded inference accelerators. We follow a late fusion approach where semantic information from multiple sen…
DPDETR: Decoupled Position Detection Transformer for Infrared-Visible Object Detection
School of Communications and Information Engineering, Chongqing University of Posts and Telecommunications（通信与信息工程学院，重庆邮电大学）；School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（智能系统工程学院，中山大学深圳校区）；School of Mathematics and Statistics, Xi’an Jiaotong University（数学与统计学院，西安交通大学）
Multispectral pedestrian detection is capable of adapting to insufficient illumination conditions by leveraging color-thermal modalities. On the other hand, it is still lacking of in-depth insights on how to fuse the two modalities effectively. Compared with traditional pedestrian detection, we find multispectral pedestrian detection suffers from modality imbalance problems which will hinder the optimization process of dual-modality network and depress the performance of detector. Inspired by this observation, we propose Modality Balance Network (MBNet) which facilitates the optimization proce…
Cross-Modal Urban Sensing: Evaluating Sound-Vision Alignment Across Street-Level and Aerial Imagery
Department of Geography, University of South Carolina（南卡罗来纳大学地理系）；Department of Environmental Sciences, Emory University（埃默里大学环境科学系）；University of Canterbury（坎特伯雷大学）；School of Resource and Environmental Sciences, Wuhan University（武汉大学资源与环境科学学院）
The Hong Kong Polytechnic University（香港理工大学）；Bavarian Forest National Park（巴伐利亚森林国家公园）
In this paper, we introduce a framework for segmenting instances of a common object class by multiple active contour evolution over semantic segmentation maps of images obtained through fully convolutional networks. The contour evolution is cast as an energy minimization problem, where the aggregate energy functional incorporates a data fit term, an explicit shape model, and accounts for object overlap. Efficient solution neighborhood operators are proposed, enabling optimization through metaheuristics such as simulated annealing. We instantiate the proposed framework in the context of segment…
Learning Image-based Tree Crown Segmentation from Enhanced Lidar-based Pseudo-labels
Department of Remote Sensing and Photogrammetry, Finnish Geospatial Research Institute（遥感与摄影测量系，芬兰地理研究机构）；Department of Computer Science, Aalto University（计算机科学系，阿alto大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Renmin University of China（中国人民大学）；CEBSIT, Chinese Academy of Sciences（中国科学院CEBSIT）
Multispectral pedestrian detection has shown great advantages under poor illumination conditions, since the thermal modality provides complementary information for the color image. However, real multispectral data suffers from the position shift problem, i.e. the color-thermal image pairs are not strictly aligned, making one object has different positions in different modalities. In deep learning based methods, this problem makes it difficult to fuse the feature maps from both modalities and puzzles the CNN training. In this paper, we propose a novel Aligned Region CNN (AR-CNN) to handle the w…
Unmanned aerial vehicles (UAV) are used in precision agriculture (PA) to enable aerial monitoring of farmlands. Intelligent methods are required to pinpoint weed infestations and make optimal choice of pesticide. UAV can fly a multispectral camera and collect data. However, the classification of multispectral images using supervised machine learning algorithms such as convolutional neural networks (CNN) requires large amount of training data. This is a common drawback in deep learning we try to circumvent making use of a semi-supervised generative adversarial networks (GAN), providing a pixel-…
MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Research Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）
Shanghai Jiao Tong University（上海交通大学）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（系统控制与信息处理教育部重点实验室）；Science and Technology on Near-Surface Detection Laboratory（近地面探测技术重点实验室）；University of Michigan-Shanghai Jiao Tong University Joint Institute（密西根大学-上海交通大学联合学院）
Thermal infrared (TIR) image has proven effectiveness in providing temperature cues to the RGB features for multispectral pedestrian detection. Most existing methods directly inject the TIR modality into the RGB-based framework or simply ensemble the results of two modalities. This, however, could lead to inferior detection performance, as the RGB and TIR features generally have modality-specific noise, which might worsen the features along with the propagation of the network. Therefore, this work proposes an effective and efficient cross-modality fusion module called Bi-directional Adaptive A…
