We propose an algorithm for the fusion of partial images collected from the visual and infrared cameras such that the visual and infrared images are the real and imaginary parts of a complex function. The proposed image fusion algorithm of the complex function is a generalization for the algorithm of conventional image addition in the same way as the addition of complex numbers is the generalization for the addition of real numbers. The proposed algorithm of the complex function is simple in use and non-demanding in computer power. The complex form of the fused image opens a possibility to form the fused image either as the amplitude image or as a phase image, which in turn can be in several forms. We show theoretically that the local contrast of the fused phase images is higher than those of the partial images as well as in comparison with the images obtained by the algorithm of the simple or weighted addition. Experimental image quality assessment of the fused phase images performed using the histograms, entropy shows the higher quality of the phase images in comparison with those of the input partial images as well as those obtained with different fusion methods reported in the literature. Keywords: digital image processing, image fusion, infrared imaging, image quality assessment
Jiangnan University（江南大学）；University of Surrey（萨里大学）
Image decomposition is a crucial subject in the field of image processing. It can extract salient features from the source image. We propose a new image decomposition method based on convolutional neural network. This method can be applied to many image processing tasks. In this paper, we apply the image decomposition network to the image fusion task. We input infrared image and visible light image and decompose them into three high-frequency feature images and a low-frequency feature image respectively. The two sets of feature images are fused using a specific fusion strategy to obtain fusion…
Infrared and visible image fusion, a hot topic in the field of image processing, aims at obtaining fused images keeping the advantages of source images. This paper proposes a novel auto-encoder (AE) based fusion network. The core idea is that the encoder decomposes an image into background and detail feature maps with low- and high-frequency information, respectively, and that the decoder recovers the original image. To this end, the loss function makes the background/detail feature maps of source images similar/dissimilar. In the test phase, background and detail feature maps are respectively…
Taiyuan University of Science and Technology（太原科技大学）；Wenzhou University（温州大学）
The existing generative adversarial fusion methods generally concatenate source images and extract local features through convolution operation, without considering their global characteristics, which tends to produce an unbalanced result and is biased towards the infrared image or visible image. Toward this end, we propose a novel end-to-end mode based on generative adversarial training to achieve better fusion balance, termed as \textit{interactive compensatory attention fusion network} (ICAFusion). In particular, in the generator, we construct a multi-level encoder-decoder network with a tr…
Infrared and visible image fusion, as a hot topic in image processing and image enhancement, aims to produce fused images retaining the detail texture information in visible images and the thermal radiation information in infrared images. A critical step for this issue is to decompose features in different scales and to merge them separately. In this paper, we propose a novel dual-stream auto-encoder (AE) based fusion network. The core idea is that the encoder decomposes an image into base and detail feature maps with low- and high-frequency information, respectively, and that the decoder is r…
Wuxi University（无锡学院）；HorizonFlow Laboratory（HorizonFlow实验室）；Jiangnan University（江南大学）；Three Gorges University（三峡大学）；Boston University（波士顿大学）；Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences（中国科学院上海光学精密机械研究所）；Shanghai University of Electric Power（上海电力大学）；Nanjing Agricultural University（南京农业大学）
Convolutional neural networks have turned into an illustrious tool for image fusion and super-resolution. However, their excellent performance cannot work without large fixed-paired datasets; and additionally, these high-demanded ground truth data always cannot be obtained easily in fusion tasks. In this study, we show that, the structures of generative networks capture a great deal of image feature priors, and then these priors are sufficient to reconstruct high-quality fused super-resolution result using only low-resolution inputs. By this way, we propose a novel self-supervised dataset-free…
Infrared and visible image fusion is an important problem in the field of image fusion which has been applied widely in many fields. To better preserve the useful information from source images, in this paper, we propose a novel image fusion method based on latent low-rank representation(LatLRR) which is simple and effective. Firstly, the source images are decomposed into low-rank parts(global structure) and salient parts(local structure) by LatLRR. Then, the low-rank parts are fused by weighted-average strategy to preserve more contour information. Then, the salient parts are simply fused by …
Infrared and visible image fusion has been a hot issue in image fusion. In this task, a fused image containing both the gradient and detailed texture information of visible images as well as the thermal radiation and highlighting targets of infrared images is expected to be obtained. In this paper, a novel Bayesian fusion model is established for infrared and visible images. In our model, the image fusion task is cast into a regression problem. To measure the variable uncertainty, we formulate the model in a hierarchical Bayesian manner. Aiming at making the fused image satisfy human visual sy…
Liaoning Normal University（辽宁师范大学）；Dalian University of Technology（大连理工大学）
General deep learning-based methods for infrared and visible image fusion rely on the unsupervised mechanism for vital information retention by utilizing elaborately designed loss functions. However, the unsupervised mechanism depends on a well designed loss function, which cannot guarantee that all vital information of source images is sufficiently extracted. In this work, we propose a novel interactive feature embedding in self-supervised learning framework for infrared and visible image fusion, attempting to overcome the issue of vital information degradation. With the help of self-supervis…
Multi-sensor fusion is widely used in the environment perception system of the autonomous vehicle. It solves the interference caused by environmental changes and makes the whole driving system safer and more reliable. In this paper, a novel visible and near-infrared fusion method based on texture information is proposed to enhance unstructured environmental images. It aims at the problems of artifact, information loss and noise in traditional visible and near infrared image fusion methods. Firstly, the structure information of the visible image (RGB) and the near infrared image (NIR) after tex…
Shanghai Jiao Tong University（上海交通大学）；Imperial College London（帝国理工学院）
Visible and infrared image fusion is one of the most important areas in image processing due to its numerous applications. While much progress has been made in recent years with efforts on developing fusion algorithms, there is a lack of code library and benchmark which can gauge the state-of-the-art. In this paper, after briefly reviewing recent advances of visible and infrared image fusion, we present a visible and infrared image fusion benchmark (VIFB) which consists of 21 image pairs, a code library of 20 fusion algorithms and 13 evaluation metrics. We also carry out large scale experiment…
Suzhou University of Science and Technology（苏州科技大学）；Jiangnan University（江南大学）
Background: Leaning redundant and complementary relationships is a critical step in the human visual system. Inspired by the infrared cognition ability of crotalinae animals, we design a joint convolution auto-encoder (JCAE) network for infrared and visible image fusion. Methods: Our key insight is to feed infrared and visible pair images into the network simultaneously and separate an encoder stream into two private branches and one common branch, the private branch works for complementary features learning and the common branch does for redundant features learning. We also build two fusion r…
SFDFusion: An Efficient Spatial-Frequency Domain Fusion Network for Infrared and Visible Image Fusion
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
Deep learning is a rapidly developing approach in the field of infrared and visible image fusion. In this context, the use of dense blocks in deep networks significantly improves the utilization of shallow information, and the combination of the Generative Adversarial Network (GAN) also improves the fusion performance of two source images. We propose a new method based on dense blocks and GANs , and we directly insert the input image-visible light image in each layer of the entire network. We use SSIM and gradient loss functions that are more consistent with perception instead of mean square e…
FusionRegister: Every Infrared and Visible Image Fusion Deserves Registration
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China（江南大学人工智能与计算机科学学院）；School of Electronic and Information Engineering, Suzhou University of Science and Technology, Suzhou, China（苏州科技大学电子与信息工程学院）
Jiangnan University（江南大学）；Suzhou University of Science and Technology（苏州科技大学）
The traditional two-state hidden Markov model divides the high frequency coefficients only into two states (large and small states). Such scheme is prone to produce an inaccurate statistical model for the high frequency subband and reduces the quality of fusion result. In this paper, a fine-grained multi-state contextual hidden Markov model (MCHMM) is proposed for infrared and visible image fusion in the non-subsampled Shearlet domain, which takes full consideration of the strong correlations and level of details of NSST coefficients. To this end, an accurate soft context variable is designed …
Jiangnan University（江南大学）；University of Surrey（萨里大学）
In the image fusion field, the design of deep learning-based fusion methods is far from routine. It is invariably fusion-task specific and requires a careful consideration. The most difficult part of the design is to choose an appropriate strategy to generate the fused image for a specific task in hand. Thus, devising learnable fusion strategy is a very challenging problem in the community of image fusion. To address this problem, a novel end-to-end fusion network architecture (RFN-Nest) is developed for infrared and visible image fusion. We propose a residual fusion network (RFN) which is bas…
CoMoFusion: Fast and High-quality Fusion of Infrared and Visible Image with Consistency Model
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；School of Electronic and Information Engineering, Suzhou University of Science and Technology（苏州科技大学电子与信息工程学院）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）
针对生成式红外与可见光图像融合训练不稳、推理慢的问题，提出 CoMoFusion，利用一致性模型在潜空间建模多模态特征并结合像素值选择损失，实现快速高质量融合，达到 SOTA 性能。
DAF-Net: A Dual-Branch Feature Decomposition Fusion Network with Domain Adaptive for Infrared and Visible Image Fusion
University of Electronic Science and Technology of China（电子科技大学）；School of Information and Communication Engineering（信息与通信工程学院）；School of Resources and Environment（资源与环境学院）
Jiangnan University（江南大学）；University of Strathclyde（斯特拉斯克莱德大学）
In this paper we propose a novel method for infrared and visible image fusion where we develop nest connection-based network and spatial/channel attention models. The nest connection-based network can preserve significant amounts of information from input data in a multi-scale perspective. The approach comprises three key elements: encoder, fusion strategy and decoder respectively. In our proposed fusion strategy, spatial attention models and channel attention models are developed that describe the importance of each spatial position and of each channel with deep features. Firstly, the source …
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）
Existing infrared and visible image fusion methods often face the dilemma of balancing modal information. Generative fusion methods reconstruct fused images by learning from data distributions, but their generative capabilities remain limited. Moreover, the lack of interpretability in modal information selection further affects the reliability and consistency of fusion results in complex scenarios. This manuscript revisits the essence of generative image fusion under the inspiration of human cognitive laws and proposes a novel infrared and visible image fusion method, termed HCLFuse. First, HC…
Infrared and visible image fusion (IVIF) expects to obtain images that retain thermal radiation information from infrared images and texture details from visible images. In this paper, a model-based convolutional neural network (CNN) model, referred to as Algorithm Unrolling Image Fusion (AUIF), is proposed to overcome the shortcomings of traditional CNN-based IVIF models. The proposed AUIF model starts with the iterative formulas of two traditional optimization models, which are established to accomplish two-scale decomposition, i.e., separating low-frequency base information and high-frequen…
Sichuan University（四川大学）；University of British Columbia（不列颠哥伦比亚大学）；Xidian University（西安电子科技大学）；Incheon National University（仁川大学）；Shandong University of Technology（山东理工大学）
In computer vision and image processing tasks, image fusion has evolved into an attractive research field. However, recent existing image fusion methods are mostly built on pixel-level operations, which may produce unacceptable artifacts and are time-consuming. In this paper, a symmetric encoder-decoder with a residual block (SEDR) for infrared and visible image fusion is proposed. For the training stage, the SEDR network is trained with a new dataset to obtain a fixed feature extractor. For the fusion stage, first, the trained model is utilized to extract the intermediate features and compens…
Jiangnan University（江南大学）；University of Surrey（萨里大学）
Image decomposition is crucial for many image processing tasks, as it allows to extract salient features from source images. A good image decomposition method could lead to a better performance, especially in image fusion tasks. We propose a multi-level image decomposition method based on latent low-rank representation(LatLRR), which is called MDLatLRR. This decomposition method is applicable to many image processing fields. In this paper, we focus on the image fusion task. We develop a novel image fusion framework based on MDLatLRR, which is used to decompose source images into detail parts(s…
MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusion
College of Computer Science and Technology, Ocean University of China（中国海洋大学计算机科学与技术学院）；College of Electronic Engineering, Ocean University of China（中国海洋大学电子工程学院）
EPOFusion: Exposure aware Progressive Optimization Method for Infrared and Visible Image Fusion
College of Information Engineering, Zhejiang University of Technology（浙江工业大学信息工程学院）；College of Information Science and Technology, Zhejiang Shuren University（浙江树人大学信息科技学院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电机电子工程系）
University of Electronic Science and Technology of China（电子科技大学）
Infrared and visible images, as multi-modal image pairs, show significant differences in the expression of the same scene. The image fusion task is faced with two problems: one is to maintain the unique features between different modalities, and the other is to maintain features at various levels like local and global features. This paper discusses the limitations of deep learning models in image fusion and the corresponding optimization strategies. Based on artificially designed structures and constraints, we divide models into explicit models, and implicit models that adaptively learn high-l…
Jiangnan University（江南大学）；University of Strathclyde（思克莱德大学）
Feature extraction and processing tasks play a key role in Image Fusion, and the fusion performance is directly affected by the different features and processing methods undertaken. By contrast, most of deep learning-based methods use deep features directly without feature extraction or processing. This leads to the fusion performance degradation in some cases. To solve these drawbacks, we propose a deep features and zero-phase component analysis (ZCA) based novel fusion framework is this paper. Firstly, the residual network (ResNet) is used to extract deep features from source images. Then ZC…
UMCFuse: A Unified Multiple Complex Scenes Infrared and Visible Image Fusion Framework
Guangdong-HongKong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology, School of Physics and Optoelectronic Engineering, Foshan University（广东-香港-澳门联合智能微纳光电技术实验室，物理与光电工程学院，佛山大学）；School of Engineering, Hong Kong University of Science and Technology（工程学院，香港科学与技术大学）；School of Information Engineering and Automation, Kunming University of Science and Technology（信息工程与自动化学院，昆明理工大学）；School of Mechanical Electronic and Information Engineering, China University of Mining and Technology（机械电子与信息工程学院，中国矿业大学）
Modality-Aware Infrared and Visible Image Fusion with Target-Aware Supervision
Independent researcher（独立研究者）；Dept. of Computer Science Baylor University（计算机科学系 巴里尔大学）；Dept. of Computer Science Engineering University of Connecticut（计算机科学工程系 佛罗里达大学）；Dept. of Computer Science New York University（计算机科学系 新 york 大学）
Infrared and Visible Image Fusion Based on Implicit Neural Representations
College of Computer Science, Beijing Information Science and Technology University（北京信息科技大学计算机学院）；School of Mathematical Sciences, Capital Normal University（首都师范大学数学学院）
Fusion or Confusion? Assessing the impact of visible-thermal image fusion for automated wildlife detection
Department of Applied Geomatics, Université de Sherbrooke（舍布鲁克大学应用地理信息学系）；Quebec Centre for Biodiversity Science, McGill University（麦吉尔大学魁北克生物多样性科学中心）；Ministère de l’Environnement, de la Lutte contre les Changements Climatiques, de la Faune et des Parcs (MELCCFP)（魁北克省环境、气候变化、野生动物和公园部）；Centre d'enseignement et de recherche en foresterie de Sainte-Foy（圣福瓦林业教学与研究中心）
