Infrared and visible images, as multi-modal image pairs, show significant differences in the expression of the same scene. The image fusion task is faced with two problems: one is to maintain the unique features between different modalities, and the other is to maintain features at various levels like local and global features. This paper discusses the limitations of deep learning models in image fusion and the corresponding optimization strategies. Based on artificially designed structures and constraints, we divide models into explicit models, and implicit models that adaptively learn high-level features or can establish global pixel associations. Ten models for comparison experiments on 21 test sets were screened. The qualitative and quantitative results show that the implicit models have more comprehensive ability to learn image features. At the same time, the stability of them needs to be improved. Aiming at the advantages and limitations to be solved by existing algorithms, we discuss the main problems of multi-modal image fusion and future research directions.
Infrared and visible image fusion is an important problem in the field of image fusion which has been applied widely in many fields. To better preserve the useful information from source images, in this paper, we propose a novel image fusion method based on latent low-rank representation(LatLRR) which is simple and effective. Firstly, the source images are decomposed into low-rank parts(global structure) and salient parts(local structure) by LatLRR. Then, the low-rank parts are fused by weighted-average strategy to preserve more contour information. Then, the salient parts are simply fused by …
Infrared and visible image fusion has been a hot issue in image fusion. In this task, a fused image containing both the gradient and detailed texture information of visible images as well as the thermal radiation and highlighting targets of infrared images is expected to be obtained. In this paper, a novel Bayesian fusion model is established for infrared and visible images. In our model, the image fusion task is cast into a regression problem. To measure the variable uncertainty, we formulate the model in a hierarchical Bayesian manner. Aiming at making the fused image satisfy human visual sy…
Liaoning Normal University（辽宁师范大学）；Dalian University of Technology（大连理工大学）
General deep learning-based methods for infrared and visible image fusion rely on the unsupervised mechanism for vital information retention by utilizing elaborately designed loss functions. However, the unsupervised mechanism depends on a well designed loss function, which cannot guarantee that all vital information of source images is sufficiently extracted. In this work, we propose a novel interactive feature embedding in self-supervised learning framework for infrared and visible image fusion, attempting to overcome the issue of vital information degradation. With the help of self-supervis…
Multi-sensor fusion is widely used in the environment perception system of the autonomous vehicle. It solves the interference caused by environmental changes and makes the whole driving system safer and more reliable. In this paper, a novel visible and near-infrared fusion method based on texture information is proposed to enhance unstructured environmental images. It aims at the problems of artifact, information loss and noise in traditional visible and near infrared image fusion methods. Firstly, the structure information of the visible image (RGB) and the near infrared image (NIR) after tex…
Suzhou University of Science and Technology（苏州科技大学）；Jiangnan University（江南大学）
Background: Leaning redundant and complementary relationships is a critical step in the human visual system. Inspired by the infrared cognition ability of crotalinae animals, we design a joint convolution auto-encoder (JCAE) network for infrared and visible image fusion. Methods: Our key insight is to feed infrared and visible pair images into the network simultaneously and separate an encoder stream into two private branches and one common branch, the private branch works for complementary features learning and the common branch does for redundant features learning. We also build two fusion r…
SFDFusion: An Efficient Spatial-Frequency Domain Fusion Network for Infrared and Visible Image Fusion
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
Deep learning is a rapidly developing approach in the field of infrared and visible image fusion. In this context, the use of dense blocks in deep networks significantly improves the utilization of shallow information, and the combination of the Generative Adversarial Network (GAN) also improves the fusion performance of two source images. We propose a new method based on dense blocks and GANs , and we directly insert the input image-visible light image in each layer of the entire network. We use SSIM and gradient loss functions that are more consistent with perception instead of mean square e…
FusionRegister: Every Infrared and Visible Image Fusion Deserves Registration
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China（江南大学人工智能与计算机科学学院）；School of Electronic and Information Engineering, Suzhou University of Science and Technology, Suzhou, China（苏州科技大学电子与信息工程学院）
Jiangnan University（江南大学）；Suzhou University of Science and Technology（苏州科技大学）
The traditional two-state hidden Markov model divides the high frequency coefficients only into two states (large and small states). Such scheme is prone to produce an inaccurate statistical model for the high frequency subband and reduces the quality of fusion result. In this paper, a fine-grained multi-state contextual hidden Markov model (MCHMM) is proposed for infrared and visible image fusion in the non-subsampled Shearlet domain, which takes full consideration of the strong correlations and level of details of NSST coefficients. To this end, an accurate soft context variable is designed …
Jiangnan University（江南大学）；University of Surrey（萨里大学）
In the image fusion field, the design of deep learning-based fusion methods is far from routine. It is invariably fusion-task specific and requires a careful consideration. The most difficult part of the design is to choose an appropriate strategy to generate the fused image for a specific task in hand. Thus, devising learnable fusion strategy is a very challenging problem in the community of image fusion. To address this problem, a novel end-to-end fusion network architecture (RFN-Nest) is developed for infrared and visible image fusion. We propose a residual fusion network (RFN) which is bas…
CoMoFusion: Fast and High-quality Fusion of Infrared and Visible Image with Consistency Model
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；School of Electronic and Information Engineering, Suzhou University of Science and Technology（苏州科技大学电子与信息工程学院）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）
针对生成式红外与可见光图像融合训练不稳、推理慢的问题，提出 CoMoFusion，利用一致性模型在潜空间建模多模态特征并结合像素值选择损失，实现快速高质量融合，达到 SOTA 性能。
Jiangnan University（江南大学）；University of Strathclyde（斯特拉斯克莱德大学）
In this paper we propose a novel method for infrared and visible image fusion where we develop nest connection-based network and spatial/channel attention models. The nest connection-based network can preserve significant amounts of information from input data in a multi-scale perspective. The approach comprises three key elements: encoder, fusion strategy and decoder respectively. In our proposed fusion strategy, spatial attention models and channel attention models are developed that describe the importance of each spatial position and of each channel with deep features. Firstly, the source …
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）
Existing infrared and visible image fusion methods often face the dilemma of balancing modal information. Generative fusion methods reconstruct fused images by learning from data distributions, but their generative capabilities remain limited. Moreover, the lack of interpretability in modal information selection further affects the reliability and consistency of fusion results in complex scenarios. This manuscript revisits the essence of generative image fusion under the inspiration of human cognitive laws and proposes a novel infrared and visible image fusion method, termed HCLFuse. First, HC…
Infrared and visible image fusion (IVIF) expects to obtain images that retain thermal radiation information from infrared images and texture details from visible images. In this paper, a model-based convolutional neural network (CNN) model, referred to as Algorithm Unrolling Image Fusion (AUIF), is proposed to overcome the shortcomings of traditional CNN-based IVIF models. The proposed AUIF model starts with the iterative formulas of two traditional optimization models, which are established to accomplish two-scale decomposition, i.e., separating low-frequency base information and high-frequen…
Sichuan University（四川大学）；University of British Columbia（不列颠哥伦比亚大学）；Xidian University（西安电子科技大学）；Incheon National University（仁川大学）；Shandong University of Technology（山东理工大学）
In computer vision and image processing tasks, image fusion has evolved into an attractive research field. However, recent existing image fusion methods are mostly built on pixel-level operations, which may produce unacceptable artifacts and are time-consuming. In this paper, a symmetric encoder-decoder with a residual block (SEDR) for infrared and visible image fusion is proposed. For the training stage, the SEDR network is trained with a new dataset to obtain a fixed feature extractor. For the fusion stage, first, the trained model is utilized to extract the intermediate features and compens…
Jiangnan University（江南大学）；University of Surrey（萨里大学）
Image decomposition is crucial for many image processing tasks, as it allows to extract salient features from source images. A good image decomposition method could lead to a better performance, especially in image fusion tasks. We propose a multi-level image decomposition method based on latent low-rank representation(LatLRR), which is called MDLatLRR. This decomposition method is applicable to many image processing fields. In this paper, we focus on the image fusion task. We develop a novel image fusion framework based on MDLatLRR, which is used to decompose source images into detail parts(s…
MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusion
College of Computer Science and Technology, Ocean University of China（中国海洋大学计算机科学与技术学院）；College of Electronic Engineering, Ocean University of China（中国海洋大学电子工程学院）
EPOFusion: Exposure aware Progressive Optimization Method for Infrared and Visible Image Fusion
College of Information Engineering, Zhejiang University of Technology（浙江工业大学信息工程学院）；College of Information Science and Technology, Zhejiang Shuren University（浙江树人大学信息科技学院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电机电子工程系）
Jiangnan University（江南大学）；University of Strathclyde（思克莱德大学）
Feature extraction and processing tasks play a key role in Image Fusion, and the fusion performance is directly affected by the different features and processing methods undertaken. By contrast, most of deep learning-based methods use deep features directly without feature extraction or processing. This leads to the fusion performance degradation in some cases. To solve these drawbacks, we propose a deep features and zero-phase component analysis (ZCA) based novel fusion framework is this paper. Firstly, the residual network (ResNet) is used to extract deep features from source images. Then ZC…
UMCFuse: A Unified Multiple Complex Scenes Infrared and Visible Image Fusion Framework
Guangdong-HongKong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology, School of Physics and Optoelectronic Engineering, Foshan University（广东-香港-澳门联合智能微纳光电技术实验室，物理与光电工程学院，佛山大学）；School of Engineering, Hong Kong University of Science and Technology（工程学院，香港科学与技术大学）；School of Information Engineering and Automation, Kunming University of Science and Technology（信息工程与自动化学院，昆明理工大学）；School of Mechanical Electronic and Information Engineering, China University of Mining and Technology（机械电子与信息工程学院，中国矿业大学）
Infrared and Visible Image Fusion Based on Implicit Neural Representations
College of Computer Science, Beijing Information Science and Technology University（北京信息科技大学计算机学院）；School of Mathematical Sciences, Capital Normal University（首都师范大学数学学院）
Fusion or Confusion? Assessing the impact of visible-thermal image fusion for automated wildlife detection
Department of Applied Geomatics, Université de Sherbrooke（舍布鲁克大学应用地理信息学系）；Quebec Centre for Biodiversity Science, McGill University（麦吉尔大学魁北克生物多样性科学中心）；Ministère de l’Environnement, de la Lutte contre les Changements Climatiques, de la Faune et des Parcs (MELCCFP)（魁北克省环境、气候变化、野生动物和公园部）；Centre d'enseignement et de recherche en foresterie de Sainte-Foy（圣福瓦林业教学与研究中心）
Infrared and Visible Image Fusion with Hierarchical Human Perception
School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications（重庆邮电大学图像认知重点实验室）
Jiangnan University（江南大学）；University of Surrey（萨里大学）
In recent years, deep learning has become a very active research tool which is used in many image processing fields. In this paper, we propose an effective image fusion method using a deep learning framework to generate a single image which contains all the features from infrared and visible images. First, the source images are decomposed into base parts and detail content. Then the base parts are fused by weighted-averaging. For the detail content, we use a deep learning network to extract multi-layer features. Using these features, we use l_1-norm and weighted-average strategy to generate se…
Shanghai Jiao Tong University（上海交通大学）；Imperial College London（帝国理工学院）
Visible and infrared image fusion is one of the most important areas in image processing due to its numerous applications. While much progress has been made in recent years with efforts on developing fusion algorithms, there is a lack of code library and benchmark which can gauge the state-of-the-art. In this paper, after briefly reviewing recent advances of visible and infrared image fusion, we present a visible and infrared image fusion benchmark (VIFB) which consists of 21 image pairs, a code library of 20 fusion algorithms and 13 evaluation metrics. We also carry out large scale experiment…
SSPFusion: A Semantic Structure-Preserving Approach for Infrared and Visible Image Fusion
School of Software Engineering, Beijing Jiaotong University（北京交通大学软件工程学院）；School of Astronautics, Beihang University（北航航天学院）；Faculty of Engineering and Information Technology, University of Melbourne（墨尔本大学工程与信息技术学院）；Ordos Laboratory, Ordos（鄂尔多斯实验室）
Bridging Human Evaluation to Infrared and Visible Image Fusion
School of Mechanical Engineering, Dalian University of Technology（大连理工大学机械工程学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology & DUT-RU International School of ISE, Dalian University of Technology（大连理工大学软件学院）；College of Information Science and Technology, Dalian Maritime University（大连海事大学信息科学与技术学院）
FusionCounting: Robust visible-infrared image fusion guided by crowd counting via multi-task learning
School of Information Science and Engineering, Yanshan University（燕山大学信息科学与工程学院）；Fusion Intelligence Laboratory, Department of Computer Science, University of Exeter（埃克塞特大学计算机科学系融合智能实验室）
GrFormer: A Novel Transformer on Grassmann Manifold for Infrared and Visible Image Fusion
School of Artificial Intelligence（人工智能学院）；Computer Science, Jiangnan University（计算机科学，江南大学）；The Centre for Vision, Speech（视觉、语音与信号处理中心）；Signal Processing, University of Surrey（信号处理，萨里大学）
Direction-aware multi-scale gradient loss for infrared and visible image fusion
Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences（光电信息处理重点实验室，中国科学院）；Shenyang Institute of Automation, Chinese Academy of Sciences（沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences, Chinese Academy of Sciences（中国科学院大学）
