This paper presents the Discrete Wavelet based fusion techniques for combining perceptually important image features. SPIHT (Set Partitioning in Hierarchical Trees) algorithm is an efficient method for lossy and lossless coding of fused image. This paper presents some modifications on the SPIHT algorithm. It is based on the idea of insignificant correlation of wavelet coefficient among the medium and high frequency sub bands. In RE-MSPIHT algorithm, wavelet coefficients are scaled prior to SPIHT coding based on the sub band importance, with the goal of minimizing the MSE.
Infrared and visible image fusion is an important problem in the field of image fusion which has been applied widely in many fields. To better preserve the useful information from source images, in this paper, we propose a novel image fusion method based on latent low-rank representation(LatLRR) which is simple and effective. Firstly, the source images are decomposed into low-rank parts(global structure) and salient parts(local structure) by LatLRR. Then, the low-rank parts are fused by weighted-average strategy to preserve more contour information. Then, the salient parts are simply fused by …
Multi-focus noisy image fusion represents an important task in the field of image fusion which generates a single, clear and focused image from all source images. In this paper, we propose a novel multi-focus noisy image fusion method based on low-rank representation (LRR) which is a powerful tool in representation learning. A multi-scale transform framework is adopted in which source images are decomposed into low frequency and high frequency coefficients, respectively. For low frequency coefficients, the fused low frequency coefficients are determined by a spatial frequency strategy, while t…
Jiangnan University（江南大学）；University of Surrey（萨里大学）
In recent years, deep learning has become a very active research tool which is used in many image processing fields. In this paper, we propose an effective image fusion method using a deep learning framework to generate a single image which contains all the features from infrared and visible images. First, the source images are decomposed into base parts and detail content. Then the base parts are fused by weighted-averaging. For the detail content, we use a deep learning network to extract multi-layer features. Using these features, we use l_1-norm and weighted-average strategy to generate se…
University of Science and Technology Beijing（北京科技大学）；Norwegian University of Science and Technology（挪威科技大学）
The general aim of multi-focus image fusion is to gather focused regions of different images to generate a unique all-in-focus fused image. Deep learning based methods become the mainstream of image fusion by virtue of its powerful feature representation ability. However, most of the existing deep learning structures failed to balance fusion quality and end-to-end implementation convenience. End-to-end decoder design often leads to unrealistic result because of its non-linear mapping mechanism. On the other hand, generating an intermediate decision map achieves better quality for the fused ima…
AD-SRF: Attack-Defense Semantic Reinforcement for Multi-Modal Image Fusion
Electronic Information School, Wuhan University（武汉大学电子信息学院）；School of Robotics, Wuhan University（武汉大学机器人学院）；Suzhou Institute, Wuhan University（武汉大学苏州研究院）
Infrared and visible image fusion has been a hot issue in image fusion. In this task, a fused image containing both the gradient and detailed texture information of visible images as well as the thermal radiation and highlighting targets of infrared images is expected to be obtained. In this paper, a novel Bayesian fusion model is established for infrared and visible images. In our model, the image fusion task is cast into a regression problem. To measure the variable uncertainty, we formulate the model in a hierarchical Bayesian manner. Aiming at making the fused image satisfy human visual sy…
National Institute of Science and Technology（国家科学技术学院）；Jadavpur University（贾达普大学）
An image fusion method based on salient features is proposed in this paper. In this work, we have concentrated on salient features of the image for fusion in order to preserve all relevant information contained in the input images and tried to enhance the contrast in fused image and also suppressed noise to a maximum extent. In our system, first we have applied a mask on two input images in order to conserve the high frequency information along with some low frequency information and stifle noise to a maximum extent. Thereafter, for identification of salience features from sources images, a lo…
CoMoFusion: Fast and High-quality Fusion of Infrared and Visible Image with Consistency Model
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；School of Electronic and Information Engineering, Suzhou University of Science and Technology（苏州科技大学电子与信息工程学院）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）
针对生成式红外与可见光图像融合训练不稳、推理慢的问题，提出 CoMoFusion，利用一致性模型在潜空间建模多模态特征并结合像素值选择损失，实现快速高质量融合，达到 SOTA 性能。
RVS College of Engineering and Technology（RVS工程技术学院）
In recent years, many research achievements are made in the medical image fusion field. Fusion is basically extraction of best of inputs and conveying it to the output. Medical Image fusion means that several of various modality image information is comprehended together to form one image to express its information. The aim of image fusion is to integrate complementary and redundant information. In this paper, a multimodal image fusion algorithm based on the dual-tree complex wavelet transform (DT-CWT) and adaptive particle swarm optimization (APSO) is proposed. Fusion is achieved through the …
FusionINN: Decomposable Image Fusion for Brain Tumor Monitoring
Technische Universität Dresden（德累斯顿工业大学）；Shandong University of Science and Technology（山东科技大学）；Shanghai Tenth People’s Hospital（上海市第十人民医院）；Tongji University Medical School（同济大学医学院）
Institute of Computing Technology Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Capital Medical University（首都医科大学）
It is necessary for clinicians to comprehensively analyze patient information from different sources. Medical image fusion is a promising approach to providing overall information from medical images of different modalities. However, existing medical image fusion approaches ignore the semantics of images, making the fused image difficult to understand. In this work, we propose a new evaluation index to measure the semantic loss of fused image, and put forward a Fusion W-Net (FW-Net) for multimodal medical image fusion. The experimental results are promising: the fused image generated by our ap…
Suzhou University of Science and Technology（苏州科技大学）；Jiangnan University（江南大学）
Background: Leaning redundant and complementary relationships is a critical step in the human visual system. Inspired by the infrared cognition ability of crotalinae animals, we design a joint convolution auto-encoder (JCAE) network for infrared and visible image fusion. Methods: Our key insight is to feed infrared and visible pair images into the network simultaneously and separate an encoder stream into two private branches and one common branch, the private branch works for complementary features learning and the common branch does for redundant features learning. We also build two fusion r…
Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；Electronic Information School, Wuhan University（武汉大学电子信息学院）
The most significant problem may be undesirable effects for the spectral signatures of fused images as well as the benefits of using fused images mostly compared to their source images were acquired at the same time by one sensor. They may or may not be suitable for the fusion of other images. It becomes therefore increasingly important to investigate techniques that allow multi-sensor, multi-date image fusion to make final conclusions can be drawn on the most suitable method of fusion. So, In this study we present a new method Segmentation Fusion method (SF) for remotely sensed images is pres…
Deep learning is a rapidly developing approach in the field of infrared and visible image fusion. In this context, the use of dense blocks in deep networks significantly improves the utilization of shallow information, and the combination of the Generative Adversarial Network (GAN) also improves the fusion performance of two source images. We propose a new method based on dense blocks and GANs , and we directly insert the input image-visible light image in each layer of the entire network. We use SSIM and gradient loss functions that are more consistent with perception instead of mean square e…
Infrared and Visible Image Fusion Based on Implicit Neural Representations
College of Computer Science, Beijing Information Science and Technology University（北京信息科技大学计算机学院）；School of Mathematical Sciences, Capital Normal University（首都师范大学数学学院）
In image fusion, images obtained from different sensors are fused to generate a single image with enhanced information. In recent years, state-of-the-art methods have adopted Convolution Neural Networks (CNNs) to encode meaningful features for image fusion. Specifically, CNN-based methods perform image fusion by fusing local features. However, they do not consider long-range dependencies that are present in the image. Transformer-based models are designed to overcome this by modeling the long-range dependencies with the help of self-attention mechanism. This motivates us to propose a novel Ima…
Multi-focus image fusion is a challenging field of study that aims to provide a completely focused image by integrating focused and un-focused pixels. Most existing methods suffer from shift variance, misregistered images, and data-dependent. In this study, we introduce a novel gradient information-based multi-focus image fusion method that is robust for the aforementioned problems. The proposed method first generates gradient images from original images by using Halftoning-Inverse Halftoning (H-IH) transform. Then, Energy of Gradient (EOG) and Standard Deviation functions are used as the focu…
Sichuan University（四川大学）；University of British Columbia（不列颠哥伦比亚大学）；Xidian University（西安电子科技大学）；Incheon National University（仁川大学）；Shandong University of Technology（山东理工大学）
In computer vision and image processing tasks, image fusion has evolved into an attractive research field. However, recent existing image fusion methods are mostly built on pixel-level operations, which may produce unacceptable artifacts and are time-consuming. In this paper, a symmetric encoder-decoder with a residual block (SEDR) for infrared and visible image fusion is proposed. For the training stage, the SEDR network is trained with a new dataset to obtain a fixed feature extractor. For the fusion stage, first, the trained model is utilized to extract the intermediate features and compens…
School of New Media and Communication, Tianjin University（新媒体与传播学院，天津大学）；School of Artificial Intelligence, Tianjin University（人工智能学院，天津大学）；Low-Altitude Intelligence Laboratory, Xiong’an National Innovation Center（低空智能实验室，雄安国家创新中心）；Xiong’an Guochuang Lantian Technology Co., Ltd.（雄安国创蓝天科技有限公司）；National University of Defense Technology（国防科技大学）
Hetman Petro Sahaidachnyi National Army Academy（赫特曼·彼得·萨海达奇尼国家军事学院）；Danylo Halytsky Lviv National Medical University（丹尼洛·哈利茨基利沃夫国立医科大学）
We propose an algorithm for the fusion of partial images collected from the visual and infrared cameras such that the visual and infrared images are the real and imaginary parts of a complex function. The proposed image fusion algorithm of the complex function is a generalization for the algorithm of conventional image addition in the same way as the addition of complex numbers is the generalization for the addition of real numbers. The proposed algorithm of the complex function is simple in use and non-demanding in computer power. The complex form of the fused image opens a possibility to for…
Remote sensing image fusion (also known as pan-sharpening) aims at generating high resolution multi-spectral (MS) image from inputs of a high spatial resolution single band panchromatic (PAN) image and a low spatial resolution multi-spectral image. Inspired by the astounding achievements of convolutional neural networks (CNNs) in a variety of computer vision tasks, in this paper, we propose a two-stream fusion network (TFNet) to address the problem of pan-sharpening. Unlike previous CNN based methods that consider pan-sharpening as a super resolution problem and perform pan-sharpening in pixel…
Image fusion produces a single fused image from a set of input images. A new method for image fusion is proposed based on Weighted Average Merging Method (WAMM) in the NonSubsampled Contourlet Transform (NSCT) domain. A performance analysis on various statistical fusion rules are also analysed both in NSCT and Wavelet domain. Analysis has been made on medical images, remote sensing images and multi focus images. Experimental results shows that the proposed method, WAMM obtained better results in NSCT domain than the wavelet domain as it preserves more edges and keeps the visual quality intact …
Multi-sensor fusion is widely used in the environment perception system of the autonomous vehicle. It solves the interference caused by environmental changes and makes the whole driving system safer and more reliable. In this paper, a novel visible and near-infrared fusion method based on texture information is proposed to enhance unstructured environmental images. It aims at the problems of artifact, information loss and noise in traditional visible and near infrared image fusion methods. Firstly, the structure information of the visible image (RGB) and the near infrared image (NIR) after tex…
Infrared and visible image fusion aims to integrate comprehensive information from multiple sources to achieve superior performances on various practical tasks, such as detection, over that of a single modality. However, most existing methods directly combined the texture details and object contrast of different modalities, ignoring the dynamic changes in reality, which diminishes the visible texture in good lighting conditions and the infrared contrast in low lighting conditions. To fill this gap, we propose a dynamic image fusion framework with a multi-modal gated mixture of local-to-global …
Xidian University（西安电子科技大学）；The University of Western Australia（西澳大学）
Convolutional neural networks have recently been used for multi-focus image fusion. However, due to the lack of labeled data for supervised training of such networks, existing methods have resorted to adding Gaussian blur in focused images to simulate defocus and generate synthetic training data with ground-truth for supervised learning. Moreover, they classify pixels as focused or defocused and leverage the results to construct the fusion weight maps which then necessitates a series of post-processing steps. In this paper, we present unsupervised end-to-end learning for directly predicting th…
FS-Diff: Semantic guidance and clarity-aware simultaneous multimodal image fusion and super-resolution
School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；School of Physics and Optoelectronic Engineering, Foshan University（佛山大学物理与光电工程学院）；School of Instrumentation Science and Optelectronics Engineering, Beihang University（北航仪器科学与光电工程学院）；School of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）
Isfahan University of Technology（伊斯法罕理工大学）；McMaster University（麦克马斯特大学）；University of Michigan（密歇根大学）
The use of 3D and stereo imaging is rapidly increasing. Compression, transmission, and processing could degrade the quality of stereo images. Quality assessment of such images is different than their 2D counterparts. Metrics that represent 3D perception by human visual system (HVS) are expected to assess stereoscopic quality more accurately. In this paper, inspired by brain sensory/motor fusion process, two stereo images are fused together. Then from every fused image two synthesized images are extracted. Effects of different distortions on statistical distributions of the synthesized images a…
The demand for pedestrian detection has created a challenging problem for various visual tasks such as image fusion. As infrared images can capture thermal radiation information, image fusion between infrared and visible images could significantly improve target detection under environmental limitations. In our project, we would approach by preprocessing our dataset with image fusion technique, then using Vision Transformer model to detect pedestrians from the fused images. During the evaluation procedure, a comparison would be made between YOLOv5 and the revised ViT model performance on our f…
Jiangnan University（江南大学）；University of Surrey（萨里大学）
Image decomposition is a crucial subject in the field of image processing. It can extract salient features from the source image. We propose a new image decomposition method based on convolutional neural network. This method can be applied to many image processing tasks. In this paper, we apply the image decomposition network to the image fusion task. We input infrared image and visible light image and decompose them into three high-frequency feature images and a low-frequency feature image respectively. The two sets of feature images are fused using a specific fusion strategy to obtain fusion…
Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment
Northwestern University（西北大学）；Northeastern University（东北大学）；University of Washington（华盛顿大学）；Hong Kong Baptist University（香港 Baptist大学）；Beijing Normal - Hong Kong Baptist University（北京师范大学-香港 Baptist大学）
SFDFusion: An Efficient Spatial-Frequency Domain Fusion Network for Infrared and Visible Image Fusion
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
Infrared and visible image fusion, a hot topic in the field of image processing, aims at obtaining fused images keeping the advantages of source images. This paper proposes a novel auto-encoder (AE) based fusion network. The core idea is that the encoder decomposes an image into background and detail feature maps with low- and high-frequency information, respectively, and that the decoder recovers the original image. To this end, the loss function makes the background/detail feature maps of source images similar/dissimilar. In the test phase, background and detail feature maps are respectively…
Degradation-Robust Fusion: An Efficient Degradation-Aware Diffusion Framework for Multimodal Image Fusion in Arbitrary Degradation Scenarios
Department of Biomedical Engineering, Hefei University of Technology（合肥工业大学生物医学工程系）；Faculty of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）
Yeshwant Mahavidyala College (SRTMU)（耶什万特·马哈维迪亚拉学院（SRTMU））
The Quality of image fusion is an essential determinant of the value of processing images fusion for many applications. Spatial and spectral qualities are the two important indexes that used to evaluate the quality of any fused image. However, the jury is still out of fused image's benefits if it compared with its original images. In addition, there is a lack of measures for assessing the objective quality of the spatial resolution for the fusion methods. Therefore, an objective quality of the spatial resolution assessment for fusion images is required. Most important details of the image are …
SSN College of Engineering（SSN工程学院）；RMD Engineering College（RMD工程学院）
Image fusion plays a vital role in medical imaging. Image fusion aims to integrate complementary as well as redundant information from multiple modalities into a single fused image without distortion or loss of information. In this research work, discrete wavelet transform (DWT)and undecimated discrete wavelet transform (UDWT)-based fusion techniques using genetic algorithm (GA)foroptimalparameter(weight)estimationinthefusionprocessareimplemented and analyzed with multi-modality brain images. The lack of shift variance while performing image fusion using DWT is addressed using UDWT. The propos…
Traditional and deep learning-based fusion methods generated the intermediate decision map to obtain the fusion image through a series of post-processing procedures. However, the fusion results generated by these methods are easy to lose some source image details or results in artifacts. Inspired by the image reconstruction techniques based on deep learning, we propose a multi-focus image fusion network framework without any post-processing to solve these problems in the end-to-end and supervised learning way. To sufficiently train the fusion model, we have generated a large-scale multi-focus …
Visualisation, Faculty of Computer Science, Technische Universit\" a t Dresden, N\" o thnitzer Stra e 46, 01187 Dresden, Germany
Image fusion helps in merging two or more images to construct a more informative single fused image. Recently, unsupervised learning based convolutional neural networks (CNN) have been utilized for different types of image fusion tasks such as medical image fusion, infrared-visible image fusion for autonomous driving as well as multi-focus and multi-exposure image fusion for satellite imagery. However, it is challenging to analyze the reliability of these CNNs for the image fusion tasks since no groundtruth is available. This led to the use of a wide variety of model architectures and optimiza…
Multimodal image fusion aims to combine relevant information from images acquired with different sensors. In medical imaging, fused images play an essential role in both standard and automated diagnosis. In this paper, we propose a novel multimodal image fusion method based on coupled dictionary learning. The proposed method is general and can be employed for different medical imaging modalities. Unlike many current medical fusion methods, the proposed approach does not suffer from intensity attenuation nor loss of critical information. Specifically, the images to be fused are decomposed into …
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；State Key Laboratory of Integrated Services Networks, Xidian University（西安电子科技大学综合业务网理论及关键技术国家重点实验室）；School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）
Infrared and visible image fusion, as a hot topic in image processing and image enhancement, aims to produce fused images retaining the detail texture information in visible images and the thermal radiation information in infrared images. A critical step for this issue is to decompose features in different scales and to merge them separately. In this paper, we propose a novel dual-stream auto-encoder (AE) based fusion network. The core idea is that the encoder decomposes an image into base and detail feature maps with low- and high-frequency information, respectively, and that the decoder is r…
Baddi University of Emerging Sciences and Technology（巴迪新兴科学与技术大学）；Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Jaypee University of Information Technology（贾佩信息技术大学）
We develop a multiexposure image fusion method based on texture features, which exploits the edge preserving and intraregion smoothing property of nonlinear diffusion filters based on partial differential equations (PDE). With the captured multiexposure image series, we first decompose images into base layers and detail layers to extract sharp details and fine details, respectively. The magnitude of the gradient of the image intensity is utilized to encourage smoothness at homogeneous regions in preference to inhomogeneous regions. Then, we have considered texture features of the base layer to…
We propose a learning framework named Feature Fusion Learning (FFL) that efficiently trains a powerful classifier through a fusion module which combines the feature maps generated from parallel neural networks. Specifically, we train a number of parallel neural networks as sub-networks, then we combine the feature maps from each sub-network using a fusion module to create a more meaningful feature map. The fused feature map is passed into the fused classifier for overall classification. Unlike existing feature fusion methods, in our framework, an ensemble of sub-network classifiers transfers i…
Institute for Human-Machine Communication（人机通信研究所）；Technical University of Munich（慕尼黑工业大学）
Acquiring spatio-temporal states of an action is the most crucial step for action classification. In this paper, we propose a data level fusion strategy, Motion Fused Frames (MFFs), designed to fuse motion information into static images as better representatives of spatio-temporal states of an action. MFFs can be used as input to any deep learning architecture with very little modification on the network. We evaluate MFFs on hand gesture recognition tasks using three video datasets - Jester, ChaLearn LAP IsoGD and NVIDIA Dynamic Hand Gesture Datasets - which require capturing long-term tempora…
This paper presents a concept of image pixel fusion of visual and thermal faces, which can significantly improve the overall performance of a face recognition system. Several factors affect face recognition performance including pose variations, facial expression changes, occlusions, and most importantly illumination changes. So, image pixel fusion of thermal and visual images is a solution to overcome the drawbacks present in the individual thermal and visual face images. Fused images are projected into eigenspace and finally classified using a multi-layer perceptron. In the experiments we ha…
Here an efficient fusion technique for automatic face recognition has been presented. Fusion of visual and thermal images has been done to take the advantages of thermal images as well as visual images. By employing fusion a new image can be obtained, which provides the most detailed, reliable, and discriminating information. In this method fused images are generated using visual and thermal face images in the first step. In the second step, fused images are projected into eigenspace and finally classified using a radial basis function neural network. In the experiments Object Tracking and Cla…
In this paper fusion of visual and thermal images in wavelet transformed domain has been presented. Here, Daubechies wavelet transform, called as D2, coefficients from visual and corresponding coefficients computed in the same manner from thermal images are combined to get fused coefficients. After decomposition up to fifth level (Level 5) fusion of coefficients is done. Inverse Daubechies wavelet transform of those coefficients gives us fused face images. The main advantage of using wavelet transform is that it is well-suited to manage different image resolution and allows the image decomposi…
