All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark
Guangdong-HongKong-Macao Joint Laboratory for Intelligent MicroNano Optoelectronic Technology（粤港澳联合智能微纳光电子技术实验室）；School of Physics and Optoelectronic Engineering, Foshan University（佛山大学物理与光电工程学院）；Guangdong Provincial Key Laboratory of Industrial Intelligent Inspection Technology（广东省工业智能检测技术重点实验室）；School of Instrumentation and Optoelectronic Engineering, Beihang University（北京航空航天大学仪器与光电工程学院）；School of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）；School of Artificial Intelligence, Optics and Electronics (i0PEN), School of Computer Science, Northwestern Polytechnical University（人工智能、光学与电子学院（i0PEN），西北工业大学计算机学院）
Multi-modality image fusion (MMIF) combines complementary information from different image modalities to provide a comprehensive and objective interpretation of scenes. However, existing fusion methods cannot resist different weather interferences in real-world scenes, limiting their practical applicability. To bridge this gap, we propose an end-to-end, unified all-weather MMIF model. Rather than focusing solely on pixel-level recovery, our method emphasizes maximizing the representation of key scene information through joint feature fusion and restoration. Specifically, we first decompose images into low-rank and sparse components, enabling effective feature separation for enhanced multi-modality perception. During feature recovery, we introduce a physically-aware clear feature prediction module, inferring variations in light transmission via illumination and reflectance. Clear features generated by the network are used to enhance salient information representation. We also construct a large-scale MMIF dataset with 100,000 image pairs comprehensively across rain, haze, and snow conditions, as well as covering various degradation levels and diverse scenes. Experimental results in both real-world and synthetic scenes demonstrate that the proposed method excels in image fusion and downstream tasks such as object detection, semantic segmentation, and depth estimation. The code is available at https://github.com/ixilai/AWFusion.
AWM-Fuse: Multi-Modality Image Fusion for Adverse Weather via Global and Local Text Perception
Guangdong-HongKong-Macao Joint Laboratory for Intelligent MicroNano Optoelectronic Technology（粤港澳联合智能微纳光电子技术联合实验室）；Foshan University（佛山大学）；School of Mechanical Electronic and Information Engineering（机械电子信息工程学院）；China University of Mining and Technology（中国矿业大学）；School of Information Engineering and Automation（信息工程与自动化学院）
CAWM-Mamba: A unified model for infrared-visible image fusion and compound adverse weather restoration
School of Physics（物理学院）；Optoelectronic Engineering, Foshan University, Foshan 528225, China（光学电子工程学院，佛山大学，佛山528225，中国）；Guangdong-HongKong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology, Foshan 528225, China（粤港澳联合智能微纳光电子技术实验室，佛山528225，中国）；School of Mechanical Electronic（机械电子学院）；Information Engineering, China University of Mining and Technology, Beijing 100083, China（信息工程学院，中国矿业大学，北京100083，中国）
CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion
Xi’an Jiaotong University（西安交通大学）；ETH Zürich（苏黎世联邦理工学院）；Northwestern Polytechnical University（西北工业大学）；Research and Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）；Harvard University（哈佛大学）；University of Würzburg（维尔茨堡大学）
Fuzhou University（福州大学）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；King’s College London（伦敦国王学院）
Both image registration and label fusion in the multi-atlas segmentation (MAS) rely on the intensity similarity between target and atlas images. However, such similarity can be problematic when target and atlas images are acquired using different imaging protocols. High-level structure information can provide reliable similarity measurement for cross-modality images when cooperating with deep neural networks (DNNs). This work presents a new MAS framework for cross-modality images, where both image registration and label fusion are achieved by DNNs. For image registration, we propose a consiste…
A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion
Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（教育部符号计算与知识工程重点实验室，吉林大学）；Institute of Digital Media, School of Electronic Engineering and Computer Science, Peking University（数字媒体研究所，电子工程与计算机科学学院，北京大学）
Multi-atlas segmentation (MAS) is a promising framework for medical image segmentation. Generally, MAS methods register multiple atlases, i.e., medical images with corresponding labels, to a target image; and the transformed atlas labels can be combined to generate target segmentation via label fusion schemes. Many conventional MAS methods employed the atlases from the same modality as the target image. However, the number of atlases with the same modality may be limited or even missing in many clinical applications. Besides, conventional MAS methods suffer from the computational burden of reg…
Modality-Aware Infrared and Visible Image Fusion with Target-Aware Supervision
Independent researcher（独立研究者）；Dept. of Computer Science Baylor University（计算机科学系 巴里尔大学）；Dept. of Computer Science Engineering University of Connecticut（计算机科学工程系 佛罗里达大学）；Dept. of Computer Science New York University（计算机科学系 新 york 大学）
DAE-Fuse: An Adaptive Discriminative Autoencoder for Multi-Modality Image Fusion
Department of Computer Science（计算机科学系）；Guangdong Provincial Key Lab of Interdisciplinary Research and Application for Data Science（广东省交叉学科研究与数据科学应用重点实验室）；Beijing Normal - Hong Kong Baptist University（北京师范大学-香港 Baptist大学）
Despite the recent advances of deep neural networks, object detection for adverse weather remains challenging due to the poor perception of some sensors in adverse weather. Instead of relying on one single sensor, multimodal fusion has been one promising approach to provide redundant detection information based on multiple sensors. However, most existing multimodal fusion approaches are ineffective in adjusting the focus of different sensors under varying detection environments in dynamic adverse weather conditions. Moreover, it is critical to simultaneously observe local and global informatio…
WeatherPrompt: Multi-modality Representation Learning for All-Weather Drone Visual Geo-Localization
School of Computer Engineering and Science, Shanghai University, China（上海大学计算机工程与科学学院）；Faculty of Science and Technology and Institute of Collaborative Innovation, University of Macau, China（澳门大学科技学院和协同创新研究院）
Recent learning-based image fusion methods have marked numerous progress in pre-registered multi-modality data, but suffered serious ghosts dealing with misaligned multi-modality data, due to the spatial deformation and the difficulty narrowing cross-modality discrepancy. To overcome the obstacles, in this paper, we present a robust cross-modality generation-registration paradigm for unsupervised misaligned infrared and visible image fusion (IVIF). Specifically, we propose a Cross-modality Perceptual Style Transfer Network (CPSTN) to generate a pseudo infrared image taking a visible image as i…
The fusion of multimodal sensor streams, such as camera, lidar, and radar measurements, plays a critical role in object detection for autonomous vehicles, which base their decision making on these inputs. While existing methods exploit redundant information in good environmental conditions, they fail in adverse weather where the sensory streams can be asymmetrically distorted. These rare "edge-case" scenarios are not represented in available datasets, and existing fusion architectures are not designed to handle them. To address this challenge we present a novel multimodal dataset acquired in o…
Multi-modality fusion is the guarantee of the stability of autonomous driving systems. In this paper, we propose a general multi-modality cascaded fusion framework, exploiting the advantages of decision-level and feature-level fusion, utilizing target position, size, velocity, appearance and confidence to achieve accurate fusion results. In the fusion process, dynamic coordinate alignment(DCA) is conducted to reduce the error between sensors from different modalities. In addition, the calculation of affinity matrix is the core module of sensor fusion, we propose an affinity loss that improves …
JOANNEUM RESEARCH Forschungsgesellschaft mbH（约阿内研究有限责任公司）；AIT Austrian Institute of Technology GmbH（奥地利技术研究所有限责任公司）；Austro Control, Österreichische Gesellschaft für Zivilluftfahrt mbH（奥斯特罗管制奥地利民用航空有限责任公司）；MeteoServe Wetterdienst GmbH（MeteoServe气象服务有限责任公司）
In air traffic management (ATM) all necessary operations (tactical planing, sector configuration, required staffing, runway configuration, routing of approaching aircrafts) rely on accurate measurements and predictions of the current weather situation. An essential basis of information is delivered by weather radar images (WXR), which, unfortunately, exhibit a vast amount of disturbances. Thus, the improvement of these datasets is the key factor for more accurate predictions of weather phenomena and weather conditions. Image processing methods based on texture analysis and geometric operators …
Shanghai Jiao Tong University（上海交通大学）；Shandong Cancer Hospital Affiliated to Shandong University（山东大学附属山东省肿瘤医院）；Deepwise AI Lab（深睿医疗人工智能实验室）
Segmentation of nasopharyngeal carcinoma (NPC) from Magnetic Resonance Images (MRI) is a crucial prerequisite for NPC radiotherapy. However, manually segmenting of NPC is time-consuming and labor-intensive. Additionally, single-modality MRI generally cannot provide enough information for its accurate delineation. Therefore, a multi-modality MRI fusion network (MMFNet) based on three modalities of MRI (T1, T2 and contrast-enhanced T1) is proposed to complete accurate segmentation of NPC. The backbone of MMFNet is designed as a multi-encoder-based network, consisting of several encoders to captu…
Robust RGB-T Tracking via Learnable Visual Fourier Prompt Fine-tuning and Modality Fusion Prompt Generation
Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University（教育区块链与智能技术重点实验室，教育部，广西师范大学）；Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University（多源信息挖掘与安全广西重点实验室，广西师范大学）
MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapters
School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）；Beijing Institute of Technology Chongqing Innovation Center（北京理工大学重庆创新中心）；National Key Laboratory for Space-Born Intelligent Information Processing（空间智能信息处理国家级重点实验室）；School of Automation, Beijing Institute of Technology（自动化学院，北京理工大学）
Improving Robustness of LiDAR-Camera Fusion Model against Weather Corruption from Fusion Strategy Perspective
Nanyang Technological University（南洋理工大学）；East China Normal University（华东师范大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；New York University（纽约大学）
RGB-infrared person re-identification is a challenging task due to the intra-class variations and cross-modality discrepancy. Existing works mainly focus on learning modality-shared global representations by aligning image styles or feature distributions across modalities, while local feature from body part and relationships between person images are largely neglected. In this paper, we propose a Dual-level (i.e., local and global) Feature Fusion (DF^2) module by learning attention for discriminative feature from local to global manner. In particular, the attention for a local feature is deter…
Supervise-assisted Multi-modality Fusion Diffusion Model for PET Restoration
Beijing Institute of Technology（北京理工大学）；Research and Development Center of Agricultural Bank of China（中国农业银行研发中心）；Peking University（北京大学）；Beijing Normal University（北京师范大学）
DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion
Xi’an Jiaotong University（西安交通大学）；Computer Vision Lab, ETH Zürich（苏黎世联邦理工学院计算机视觉实验室）；Northwestern Polytechnical University（西北工业大学）；Macau University of Science and Technology（澳门科技大学）；University of Würzburg（维尔茨堡大学）
Scene depth information can help visual information for more accurate semantic segmentation. However, how to effectively integrate multi-modality information into representative features is still an open problem. Most of the existing work uses DCNNs to implicitly fuse multi-modality information. But as the network deepens, some critical distinguishing features may be lost, which reduces the segmentation performance. This work proposes a unified and efficient feature selectionand-fusion network (FSFNet), which contains a symmetric cross-modality residual fusion module used for explicit fusion o…
Asynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations
Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Digital Medical Research Center, School of Basic Medical Science, Fudan University（复旦大学基础医学院数字医学研究中心）；University of South Carolina（南卡罗来纳大学）
Heudiasyc（Heudiasyc实验室）；CNRS（法国国家科学研究中心）；Université de technologie de Compiègne（贡比涅技术大学）；Quantif（Quantif团队）；LITIS（LITIS实验室）；University of Rouen Normandy（鲁昂诺曼底大学）；Institut universitaire de France（法国大学研究院）；Henri Becquerel Cancer Center（亨利·贝克勒尔癌症中心）
As information sources are usually imperfect, it is necessary to take into account their reliability in multi-source information fusion tasks. In this paper, we propose a new deep framework allowing us to merge multi-MR image segmentation results using the formalism of Dempster-Shafer theory while taking into account the reliability of different modalities relative to different classes. The framework is composed of an encoder-decoder feature extraction module, an evidential segmentation module that computes a belief function at each voxel for each modality, and a multi-modality evidence fusion…
University of Sydney（悉尼大学）；Royal Prince Alfred Hospital（阿尔弗雷德王子医院）
The analysis of multi-modality positron emission tomography and computed tomography (PET-CT) images for computer aided diagnosis applications requires combining the sensitivity of PET to detect abnormal regions with anatomical localization from CT. Current methods for PET-CT image analysis either process the modalities separately or fuse information from each modality based on knowledge about the image analysis task. These methods generally do not consider the spatially varying visual characteristics that encode different information across the different modalities, which have different priori…
When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation
Central China Normal University（华中师范大学）；Huazhong University of Science and Technology（华中科技大学）；Guangzhou University of Chinese Medicine（广州中医药大学）；The First Affiliated Hospital of Guangzhou University of Chinese Medicine（广州中医药大学第一附属医院）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
DinoRADE: Full Spectral Radar-Camera Fusion with Vision Foundation Model Features for Multi-class Object Detection in Adverse Weather
Infineon Technologies AG, Austria（英飞凌科技公司，奥地利）；Graz University of Technology, Austria（格拉茨技术大学，奥地利）；Virtual Vehicle Research GmbH, Austria（虚拟车辆研究有限公司，奥地利）
