A Multiple Source Hourglass Deep Network for Multi-Focus Image Fusion
Centro de Informática (CIn), Universidade Federal de Pernambuco (UFPE)（伯南布哥联邦大学信息中心）；Center for Advanced Methods in Biological Image Analysis (CAMBIA), California Institute of Technology（加州理工学院生物图像分析先进方法中心）
Fidel Alejandro Guerrero Peña, Pedro Diamel Marrero Fernández, Tsang Ing Ren, Germano Crispim Vasconcelos, Alexandre Cunha
Multi-Focus Image Fusion seeks to improve the quality of an acquired burst of images with different focus planes. For solving the task, an activity level measurement and a fusion rule are typically established to select and fuse the most relevant information from the sources. However, the design of this kind of method by hand is really hard and sometimes restricted to solution spaces where the optimal all-in-focus images are not contained. Then, we propose here two fast and straightforward approaches for image fusion based on deep neural networks. Our solution uses a multiple source Hourglass architecture trained in an end-to-end fashion. Models are data-driven and can be easily generalized for other kinds of fusion problems. A segmentation approach is used for recognition of the focus map, while the weighted average rule is used for fusion. We designed a training loss function for our regression-based fusion function, which allows the network to learn both the activity level measurement and the fusion rule. Experimental results show our approach has comparable results to the state-of-the-art methods with a 60X increase of computational efficiency for 520X520 resolution images.
Shenyang Aerospace University（沈阳航空航天大学）；Shenyang Fire Science and Technology Research Institute of MEM（沈阳消防科学研究所）
Rain streaks bring serious blurring and visual quality degradation, which often vary in size, direction and density. Current CNN-based methods achieve encouraging performance, while are limited to depict rain characteristics and recover image details in the poor visibility environment. To address these issues, we present a Multi-scale Hourglass Hierarchical Fusion Network (MH2F-Net) in end-to-end manner, to exactly captures rain streak features with multi-scale extraction, hierarchical distillation and information aggregation. For better extracting the features, a novel Multi-scale Hourglass E…
Multi-focus noisy image fusion represents an important task in the field of image fusion which generates a single, clear and focused image from all source images. In this paper, we propose a novel multi-focus noisy image fusion method based on low-rank representation (LRR) which is a powerful tool in representation learning. A multi-scale transform framework is adopted in which source images are decomposed into low frequency and high frequency coefficients, respectively. For low frequency coefficients, the fused low frequency coefficients are determined by a spatial frequency strategy, while t…
Due to the confined focal length of optical sensors, focusing all objects in a scene with a single sensor is a difficult task. To handle such a situation, image fusion methods are used in multi-focus environment. Discrete Cosine Transform (DCT) is a widely used image compression transform, image fusion in DCT domain is an efficient method. This paper presents a low complexity approach for multi-focus image fusion and its VLSI implementation using DCT. The proposed method is evaluated using reference/non-reference fusion measure criteria and the obtained results asserts it's effectiveness. The …
Brno University of Technology（布尔诺理工大学）；EUP Mataró, Tecnocampus（马塔罗EUP，Tecnocampus）
This paper proposes a novel algorithm for multi-focus thermal image fusion. The algorithm is based on local activity analysis and advanced pre-selection of images into fusion process. The algorithm improves the object temperature measurement error up to 5 Celsius degrees. The proposed algorithm is evaluated by half total error rate, root mean squared error, cross correlation and visual inspection. To the best of our knowledge, this is the first work devoted to multi-focus thermal image fusion. For testing of proposed algorithm we acquire six thermal image set with objects at different focal de…
Multi-focus image fusion is a challenging field of study that aims to provide a completely focused image by integrating focused and un-focused pixels. Most existing methods suffer from shift variance, misregistered images, and data-dependent. In this study, we introduce a novel gradient information-based multi-focus image fusion method that is robust for the aforementioned problems. The proposed method first generates gradient images from original images by using Halftoning-Inverse Halftoning (H-IH) transform. Then, Energy of Gradient (EOG) and Standard Deviation functions are used as the focu…
In this paper we propose a very efficient method to fuse the unregistered multi-focus microscopical images based on the speed-up robust features (SURF). Our method follows the pipeline of first registration and then fusion. However, instead of treating the registration and fusion as two completely independent stage, we propose to reuse the determinant of the approximate Hessian generated in SURF detection stage as the corresponding salient response for the final image fusion, thus it enables nearly cost-free saliency map generation. In addition, due to the adoption of SURF scale space represen…
State Key Laboratory of Ocean Sensing & Ocean College, Zhejiang University（海洋传感国家重点实验室与海洋学院，浙江大学）；Donghai Laboratory（东海实验室）；Hainan Institute, Zhejiang University（海南学院，浙江大学）；Hainan Provincial Observatory of Ecological Environment and Fishery Resource in Yazhou Bay（三亚市生态环境与渔业资源省级观测站）
School of IoT Engineering, Jiangnan University（江南大学物联网工程学院）；College of Mathematics and Information Science, East China Institute of Technology（东华理工大学数学与信息科学学院）
This paper presents an improved dual channel pulse coupled neural network (IDC-PCNN) model for image fusion. The model can overcome some defects of standard PCNN model. In this fusion scheme, the multiplication rule is replaced by addition rule in the information fusion pool of dual channel PCNN (DC-PCNN) model. Meanwhile the sum of modified Laplacian (SML) measure is adopted, which is better than other focus measures. This method not only inherits the good characteristics of the standard PCNN model but also enhances the computing efficiency and fusion quality. The performance of the proposed …
University of Science and Technology Beijing（北京科技大学）；Norwegian University of Science and Technology（挪威科技大学）
The general aim of multi-focus image fusion is to gather focused regions of different images to generate a unique all-in-focus fused image. Deep learning based methods become the mainstream of image fusion by virtue of its powerful feature representation ability. However, most of the existing deep learning structures failed to balance fusion quality and end-to-end implementation convenience. End-to-end decoder design often leads to unrealistic result because of its non-linear mapping mechanism. On the other hand, generating an intermediate decision map achieves better quality for the fused ima…
Since it is usually difficult to capture an all-in-focus image of a 3D scene directly, various multi-focus image fusion methods are employed to generate it from several images focusing at different depths. However, the performance of existing methods is barely satisfactory and often degrades for areas near the focused/defocused boundary (FDB). In this paper, a boundary aware method using deep neural network is proposed to overcome this problem. (1) Aiming to acquire improved fusion images, a 2-channel deep network is proposed to better extract the relative defocus information of the two source…
Xidian University（西安电子科技大学）；The University of Western Australia（西澳大学）
Convolutional neural networks have recently been used for multi-focus image fusion. However, due to the lack of labeled data for supervised training of such networks, existing methods have resorted to adding Gaussian blur in focused images to simulate defocus and generate synthetic training data with ground-truth for supervised learning. Moreover, they classify pixels as focused or defocused and leverage the results to construct the fusion weight maps which then necessitates a series of post-processing steps. In this paper, we present unsupervised end-to-end learning for directly predicting th…
Image fusion is one of the recent trends in image registration which is an essential field of image processing. The basic principle of this paper is to fuse multi-focus images using simple statistical standard deviation. Firstly, the simple standard deviation for the k-by-k window inside each of the multi-focus images was computed. The contribution in this paper came from the idea that the focused part inside an image had high details rather than the unfocused part. Hence, the dispersion between pixels inside the focused part is higher than the dispersion inside the unfocused part. Secondly, a…
Babol Noshirvani University of Technology（巴博勒努希尔瓦尼理工大学）
The purpose of multi-focus image fusion is gathering the essential information and the focused parts from the input multi-focus images into a single image. These multi-focused images are captured with different depths of focus of cameras. Multi-focus image fusion is very time-saving and appropriate in discrete cosine transform (DCT) domain, especially when JPEG images are used in visual sensor networks (VSN). The previous works in DCT domain have some errors in selection of the suitable divided blocks according to their criterion for measurement of the block contrast. In this paper, we used va…
University of Science and Technology Beijing（北京科技大学）；Beijing Advanced Innovation Center for Materials Genome Engineering（北京材料基因工程高精尖创新中心）；School of Computer and Communication Engineering（计算机与通信工程学院）；Beijing Key Laboratory of Knowledge Engineering for Materials Science（北京市材料科学知识工程重点实验室）；Institute for Advanced Materials and Technology（先进材料与技术研究所）
In this work, we propose a novel unsupervised deep learning model to address multi-focus image fusion problem. First, we train an encoder-decoder network in unsupervised manner to acquire deep feature of input images. And then we utilize these features and spatial frequency to measure activity level and decision map. Finally, we apply some consistency verification methods to adjust the decision map and draw out fused result. The key point behind of proposed method is that only the objects within the depth-of-field (DOF) have sharp appearance in the photograph while other objects are likely to …
UHD-MFF: Shattering Barriers in Multi-Focus Ultra-High-Definition Image Fusion via Learnable Lookup Tables
Electronic Information School, Wuhan University（武汉大学电子信息学院）；School of Robotics, Wuhan University（武汉大学机器人学院）；School of Automation, Southeast University（东南大学自动化学院）
Multi-Focus Image Fusion (MFIF) is a promising image enhancement technique to obtain all-in-focus images meeting visual needs and it is a precondition of other computer vision tasks. One of the research trends of MFIF is to avoid the defocus spread effect (DSE) around the focus/defocus boundary (FDB). In this paper,we propose a network termed MFIF-GAN to attenuate the DSE by generating focus maps in which the foreground region are correctly larger than the corresponding objects. The Squeeze and Excitation Residual module is employed in the network. By combining the prior knowledge of training …
ReDiffuse: Rotation Equivariant Diffusion Model for Multi-focus Image Fusion
School of Computer Science and Technology, Xi'an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Information Engineering, Chang'an University（长安大学信息工程学院）
Among the representation learning, the low-rank representation (LRR) is one of the hot research topics in many fields, especially in image processing and pattern recognition. Although LRR can capture the global structure, the ability of local structure preservation is limited because LRR lacks dictionary learning. In this paper, we propose a novel multi-focus image fusion method based on dictionary learning and LRR to get a better performance in both global and local structure. Firstly, the source images are divided into several patches by sliding window technique. Then, the patches are classi…
In recent years, various applications in computer vision have achieved substantial progress based on deep learning, which has been widely used for image fusion and shown to achieve adequate performance. However, suffering from limited ability in modeling the spatial correspondence of different source images, it still remains a great challenge for existing unsupervised image fusion models to extract appropriate feature and achieves adaptive and balanced fusion. In this paper, we propose a novel cross-attention-guided image fusion network, which is a unified and unsupervised framework for multi-…
Traditional and deep learning-based fusion methods generated the intermediate decision map to obtain the fusion image through a series of post-processing procedures. However, the fusion results generated by these methods are easy to lose some source image details or results in artifacts. Inspired by the image reconstruction techniques based on deep learning, we propose a multi-focus image fusion network framework without any post-processing to solve these problems in the end-to-end and supervised learning way. To sufficiently train the fusion model, we have generated a large-scale multi-focus …
We address the multi-focus image fusion problem, where multiple images captured with different focal settings are to be fused into an all-in-focus image of higher quality. Algorithms for this problem necessarily admit the source image characteristics along with focused and blurred features. However, most sparsity-based approaches use a single dictionary in focused feature space to describe multi-focus images, and ignore the representations in blurred feature space. We propose a multi-focus image fusion approach based on sparse representation using a coupled dictionary. It exploits the observat…
Neurodynamics-Driven Coupled Neural P Systems for Multi-Focus Image Fusion
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；Ministry of Education Key Laboratory of Intelligent Networks and Network Security, China（教育部智能网络与网络安全重点实验室）；Shaanxi Province Key Laboratory of Big Data Knowledge Engineering（陕西省大数据知识工程重点实验室）；Chang’an University（长安大学）
Image fusion is a technique to integrate information from multiple source images with complementary information to improve the richness of a single image. Due to insufficient task-specific training data and corresponding ground truth, most existing end-to-end image fusion methods easily fall into overfitting or tedious parameter optimization processes. Two-stage methods avoid the need of large amount of task-specific training data by training encoder-decoder network on large natural image datasets and utilizing the extracted features for fusion, but the domain gap between natural images and di…
School of Mechatronic Engineering and Automation, Shanghai University（上海大学机电工程与自动化学院）；Department of Mechanical and Industrial Engineering, University of Toronto（多伦多大学机械与工业工程系）；Shanghai University（上海大学）；Department of Mechanical and Industrial Engineering, University of Toronto, Canada（多伦多大学机械与工业工程系）
Multi-focus image fusion is a technique for obtaining an all-in-focus image in which all objects are in focus to extend the limited depth of field (DoF) of an imaging system. Different from traditional RGB-based methods, this paper presents a new multi-focus image fusion method assisted by depth sensing. In this work, a depth sensor is used together with a color camera to capture images of a scene. A graph-based segmentation algorithm is used to segment the depth map from the depth sensor, and the segmented regions are used to guide a focus algorithm to locate in-focus image blocks from among …
Image fusion in visual sensor networks (VSNs) aims to combine information from multiple images of the same scene in order to transform a single image with more information. Image fusion methods based on discrete cosine transform (DCT) are less complex and time-saving in DCT based standards of image and video which makes them more suitable for VSN applications. In this paper, an efficient algorithm for the fusion of multi-focus images in the DCT domain is proposed. The Sum of modified laplacian (SML) of corresponding blocks of source images is used as a contrast criterion and blocks with the la…
Human pose estimation (HPE) is a classical task in computer vision that focuses on representing the orientation of a person by identifying the positions of their joints. We design a lighterversion of the stacked hourglass network with minimal loss in performance of the model. The lightweight 2-stacked hourglass has a reduced number of channels with depthwise separable convolutions, residual connections with concatenation, and residual connections between the necks of the hourglasses. The final model has a marginal drop in performance with 79% reduction in the number of parameters and a similar…
Beijing Normal University（北京师范大学）；Xidian University（西安电子科技大学）；Xinjiang Medical University（新疆医科大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Recently, deep-learning-based super-resolution methods have achieved excellent performances, but mainly focus on training a single generalized deep network by feeding numerous samples. Yet intuitively, each image has its representation, and is expected to acquire an adaptive model. For this issue, we propose a novel image-specific convolutional kernel modulation (IKM) by exploiting the global contextual information of image or feature to generate an attention weight for adaptively modulating the convolutional kernels, which outperforms the vanilla convolution and several existing attention mec…
The paper addresses the image fusion problem, where multiple images captured with different focus distances are to be combined into a higher quality all-in-focus image. Most current approaches for image fusion strongly rely on the unrealistic noise-free assumption used during the image acquisition, and then yield limited robustness in fusion processing. In our approach, we formulate the multi-focus image fusion problem in terms of an analysis sparse model, and simultaneously perform the restoration and fusion of multi-focus images. Based on this model, we propose an analysis operator learning,…
School of IoT Engineering, Jiangnan University（江南大学物联网工程学院）
A novel multi-focus image fusion algorithm performed in spatial domain based on similarity characteristics is proposed incorporating with region segmentation. In this paper, a new similarity measure is developed based on the structural similarity (SSIM) index, which is more suitable for multi-focus image segmentation. Firstly, the SSNSIM map is calculated between two input images. Then we segment the SSNSIM map using watershed method, and merge the small homogeneous regions with fuzzy c-means clustering algorithm (FCM). For three source images, a joint region segmentation method based on segme…
Jiangnan University（江南大学）；University of Surrey（萨里大学）
Image decomposition is crucial for many image processing tasks, as it allows to extract salient features from source images. A good image decomposition method could lead to a better performance, especially in image fusion tasks. We propose a multi-level image decomposition method based on latent low-rank representation(LatLRR), which is called MDLatLRR. This decomposition method is applicable to many image processing fields. In this paper, we focus on the image fusion task. We develop a novel image fusion framework based on MDLatLRR, which is used to decompose source images into detail parts(s…
Multi-focus image fusion (MFIF) has attracted considerable interests due to its numerous applications. While much progress has been made in recent years with efforts on developing various MFIF algorithms, some issues significantly hinder the fair and comprehensive performance comparison of MFIF methods, such as the lack of large-scale test set and the random choices of objective evaluation metrics in the literature. To solve these issues, this paper presents a multi-focus image fusion benchmark (MFIFB) which consists a test set of 105 image pairs, a code library of 30 MFIF algorithms, and 20 e…
Multi-focus image fusion (MFF) is a fundamental task in the field of computational photography. Current methods have achieved significant performance improvement. It is found that current methods are evaluated on simulated image sets or Lytro dataset. Recently, a growing number of researchers pay attention to defocus spread effect, a phenomenon of real-world multi-focus images. Nonetheless, defocus spread effect is not obvious in simulated or Lytro datasets, where popular methods perform very similar. To compare their performance on images with defocus spread effect, this paper constructs a ne…
Xi’an Jiaotong University（西安交通大学）；University of Virginia（弗吉尼亚大学）；Hikvision（海康威视）
Multi-focus image fusion (MFF) is a popular technique to generate an all-in-focus image, where all objects in the scene are sharp. However, existing methods pay little attention to defocus spread effects of the real-world multi-focus images. Consequently, most of the methods perform badly in the areas near focus map boundaries. According to the idea that each local region in the fused image should be similar to the sharpest one among source images, this paper presents an optimization-based approach to reduce defocus spread effects. Firstly, a new MFF assessmentmetric is presented by combining …
In order to solve the robustness and generality problems of the image fusion task,inspired by the human brain cognitive mechanism, we propose a robust and general image fusion method with autonomous evolution ability, and is therefore denoted with AE-Net. Through the collaborative optimization of multiple image fusion methods to simulate the cognitive process of human brain, unsupervised learning image fusion task can be transformed into semi-supervised image fusion task or supervised image fusion task, thus promoting the evolutionary ability of network model weight. Firstly, the relationship …
École polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
We propose a real-time image fusion method using pre-trained neural networks. Our method generates a single image containing features from multiple sources. We first decompose images into a base layer representing large scale intensity variations, and a detail layer containing small scale changes. We use visual saliency to fuse the base layers, and deep feature maps extracted from a pre-trained neural network to fuse the detail layers. We conduct ablation studies to analyze our method's parameters such as decomposition filters, weight construction methods, and network depth and architecture. T…
Jiangnan University（江南大学）；HorizonFlow Laboratory（HorizonFlow实验室）；Chongqing Three Gorges University（重庆三峡大学）；Boston University（波士顿大学）；Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences（中国科学院上海光学精密机械研究所）；OptiX+ Laboratory（OptiX+实验室）
Multi-focus image fusion (MFIF) and super-resolution (SR) are the inverse problem of imaging model, purposes of MFIF and SR are obtaining all-in-focus and high-resolution 2D mapping of targets. Though various MFIF and SR methods have been designed; almost all the them deal with MFIF and SR separately. This paper unifies MFIF and SR problems in the physical perspective as the multi-focus image super resolution fusion (MFISRF), and we propose a novel unified dataset-free unsupervised framework named deep fusion prior (DFP) based-on deep image prior (DIP) to address such MFISRF with single model.…
Jiangnan University（江南大学）；University of Surrey（萨里大学）
Deep learning-based image fusion approaches have obtained wide attention in recent years, achieving promising performance in terms of visual perception. However, the fusion module in the current deep learning-based methods suffers from two limitations, \textit{i.e.}, manually designed fusion function, and input-independent network learning. In this paper, we propose an unsupervised adaptive image fusion method to address the above issues. We propose a feature mutual mapping fusion module and dual-branch multi-scale autoencoder. More specifically, we construct a global map to measure the connec…
FFCA-Net: Stereo Image Compression via Fast Cascade Alignment of Side Information
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Research Center of Artificial Intelligence, Peng Cheng Laboratory（鹏城实验室人工智能研究中心）
The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In this paper, we propose to incorporate convolutional neural networks with a multi-context attention mechanism into an end-to-end framework for human pose estimation. We adopt stacked hourglass networks to generate attention maps from features at multiple resolutions with various semantics. The Conditional Random Field (CRF) is utilized to model the correlations among neighboring regions in the attention map. We further combine the holistic attention model, which focuses on the global consistency of the full human body, and the body part attention model, which focuses on the detailed descript…
Image fusion produces a single fused image from a set of input images. A new method for image fusion is proposed based on Weighted Average Merging Method (WAMM) in the NonSubsampled Contourlet Transform (NSCT) domain. A performance analysis on various statistical fusion rules are also analysed both in NSCT and Wavelet domain. Analysis has been made on medical images, remote sensing images and multi focus images. Experimental results shows that the proposed method, WAMM obtained better results in NSCT domain than the wavelet domain as it preserves more edges and keeps the visual quality intact …
Insufficient labeled training datasets is one of the bottlenecks of 3D hand pose estimation from monocular RGB images. Synthetic datasets have a large number of images with precise annotations, but the obvious difference with real-world datasets impacts the generalization. Little work has been done to bridge the gap between two domains over their wide difference. In this paper, we propose a domain adaptation method called Adaptive Wasserstein Hourglass (AW Hourglass) for weakly-supervised 3D hand pose estimation, which aims to distinguish the difference and explore the common characteristics (…
Human pose estimation (HPE) is one of the most challenging tasks in computer vision as humans are deformable by nature and thus their pose has so much variance. HPE aims to correctly identify the main joint locations of a single person or multiple people in a given image or video. Locating joints of a person in images or videos is an important task that can be applied in action recognition and object tracking. As have many computer vision tasks, HPE has advanced massively with the introduction of deep learning to the field. In this paper, we focus on one of the deep learning-based approaches o…
FusionINN: Decomposable Image Fusion for Brain Tumor Monitoring
Technische Universität Dresden（德累斯顿工业大学）；Shandong University of Science and Technology（山东科技大学）；Shanghai Tenth People’s Hospital（上海市第十人民医院）；Tongji University Medical School（同济大学医学院）
