Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models
Central South University（中南大学）；Changsha University of Science and Technology（长沙理工大学）；Hunan University（湖南大学）；Peng Cheng Laboratory（鹏城实验室）；Beihang University（北京航空航天大学）；Wuhan University（武汉大学）
Jun Yue, Leyuan Fang, Shaobo Xia, Yue Deng, Jiayi Ma
备注：
This work has been submitted to the IEEE for possible publication
Color plays an important role in human visual perception, reflecting the spectrum of objects. However, the existing infrared and visible image fusion methods rarely explore how to handle multi-spectral/channel data directly and achieve high color fidelity. This paper addresses the above issue by proposing a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data, which increases the ability of multi-source information aggregation and the fidelity of colors. In specific, instead of converting multi-channel images into single-channel data in existing fusion methods, we create the multi-channel data distribution with a denoising network in a latent space with forward and reverse diffusion process. Then, we use the the denoising network to extract the multi-channel diffusion features with both visible and infrared information. Finally, we feed the multi-channel diffusion features to the multi-channel fusion module to directly generate the three-channel fused image. To retain the texture and intensity information, we propose multi-channel gradient loss and intensity loss. Along with the current evaluation metrics for measuring texture and intensity fidelity, we introduce a new evaluation metric to quantify color fidelity. Extensive experiments indicate that our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity.
SGDFuse: SAM-Guided Diffusion Model for High-Fidelity Infrared and Visible Image Fusion
School of Computer Science（计算机科学学院）；Business University, Yantai, China（烟台商学院）；College of Computer and Data Science（计算机与数据科学学院）；Fuzhou University, Fuzhou, China（福州大学）；School of Computing and Mathematical Sciences（计算与数学科学学院）；University of Leicester, Leicester, United Kingdom（莱斯特大学）；School of Computing and Artificial Intelligence（计算与人工智能学院）；Director of the Rapid-Rich Object Search Laboratory（快速丰富对象搜索实验室主任）；NTU-PKU Joint Research Institute, Nanyang Technological University, Singapore（NTU-PKU联合研究机构，新加坡国立大学）
HKUST（香港科技大学）；SenseTime Research（商汤科技研究院）；Centre for Artificial Intelligence and Robotics（人工智能与机器人中心）
We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views while preserving specific details of the input image. High-fidelity 3D GAN inversion is inherently challenging due to the geometry-texture trade-off in 3D inversion, where overfitting to a single view input image often damages the estimated geometry during the latent optimization. To solve this challenge, we propose a novel pipeline that builds on the pseudo-multi-view estimation with visibility analysis. We keep the original textures for the visible parts and …
VI-Diff: Unpaired Visible-Infrared Translation Diffusion Model for Single Modality Labeled Visible-Infrared Person Re-identification
University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；The Center for Research on Intelligent Perception and Computing (CRIPAC)（智能感知与计算研究中心）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）
TC-PDM: Temporally Consistent Patch Diffusion Models for Infrared-to-Visible Video Translation
Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Safran Electronics and Defense Australasia（赛峰电子与防务澳大拉西亚分公司）；Department of Data Science and Artificial Intelligence, Monash University（莫纳什大学数据科学与人工智能系）
Thermal to visible face verification is a challenging problem due to the large domain discrepancy between the modalities. Existing approaches either attempt to synthesize visible faces from thermal faces or extract robust features from these modalities for cross-modal matching. In this paper, we take a different approach in which we make use of the attributes extracted from the visible image to synthesize the attribute-preserved visible image from the input thermal image for cross-modal matching. A pre-trained VGG-Face network is used to extract the attributes from the visible image. Then, a n…
Johns Hopkins University（约翰斯·霍普金斯大学）；U.S. Army CCDC Army Research Laboratory（美国陆军CCDC陆军研究实验室）
Polarimetric thermal to visible face verification entails matching two images that contain significant domain differences. Several recent approaches have attempted to synthesize visible faces from thermal images for cross-modal matching. In this paper, we take a different approach in which rather than focusing only on synthesizing visible faces from thermal faces, we also propose to synthesize thermal faces from visible faces. Our intuition is based on the fact that thermal images also contain some discriminative information about the person for verification. Deep features from a pre-trained C…
Rutgers, The State University of New Jersey（罗格斯大学，新泽西州立大学）；U.S. Army Research Laboratory（美国陆军研究实验室）
The large domain discrepancy between faces captured in polarimetric (or conventional) thermal and visible domain makes cross-domain face recognition quite a challenging problem for both human-examiners and computer vision algorithms. Previous approaches utilize a two-step procedure (visible feature estimation and visible image reconstruction) to synthesize the visible image given the corresponding polarimetric thermal image. However, these are regarded as two disjoint steps and hence may hinder the performance of visible face reconstruction. We argue that joint optimization would be a better w…
FaceRefiner: High-Fidelity Facial Texture Refinement with Differentiable Rendering-based Style Transfer
School of Informatics, Xiamen University（厦门大学信息学院）；China Mobile (Hangzhou) Information Technology Co., Ltd.（中国移动（杭州）信息技术有限公司）；School of Computing and Data Science, Xiamen University Malaysia（马来西亚厦门大学计算与数据科学学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
We qualitatively examine the accuracy and fidelity between two diffusion-based image generation systems, namely DALL-E 2 and Luna, which have massive differences in training datasets, algorithmic approaches, prompt resolvement, and output upscaling. The methodology used is a qualitative benchmark created by Saharia et al. and in our research we conclude that DALL-E 2 significantly edges Luna in both alignment and fidelity comparisons.
Bridging global context interactions correctly is important for high-fidelity image completion with large masks. Previous methods attempting this via deep or large receptive field (RF) convolutions cannot escape from the dominance of nearby interactions, which may be inferior. In this paper, we propose to treat image completion as a directionless sequence-to-sequence prediction task, and deploy a transformer to directly capture long-range dependence in the encoder. Crucially, we employ a restrictive CNN with small and non-overlapping RF for weighted token representation, which allows the trans…
Rutgers, The State University of New Jersey（罗格斯大学新泽西州立大学）；U.S. Army Research Laboratory（美国陆军研究实验室）；Booz Allen Hamilton（博思艾伦汉密尔顿公司）；Johns Hopkins University（约翰斯·霍普金斯大学）
The large domain discrepancy between faces captured in polarimetric (or conventional) thermal and visible domain makes cross-domain face verification a highly challenging problem for human examiners as well as computer vision algorithms. Previous approaches utilize either a two-step procedure (visible feature estimation and visible image reconstruction) or an input-level fusion technique, where different Stokes images are concatenated and used as a multi-channel input to synthesize the visible image given the corresponding polarimetric signatures. Although these methods have yielded improvemen…
Improved Sub-Visible Particle Classification in Flow Imaging Microscopy via Generative AI-Based Image Synthesis
Center for Biosystems and Biotech Data Science（生物系统与生物技术数据科学中心）；Ghent University Global Campus（根特大学全球校区）；Department of Electronics and Information Systems（电子与信息系统系）；Faculty of Pharmaceutical Sciences（药学系）；Biopharmaceutical Technology, TUM School of Life Sciences（生物制药技术，技术大学生命科学学院）；Department of Mathematics, Computer Science and Statistics（数学、计算机科学与统计学系）
CapHDR2IR: Caption-Driven Transfer from Visible Light to Infrared Domain
WMG, University of Warwick（华威大学制造工程学院）；School of Information Science and Technology, East China University of Science and Technology（华东理工大学信息科学与工程学院）；School of Engineering, Newcastle University（纽卡斯尔大学工程学院）
Johns Hopkins University（约翰斯·霍普金斯大学）；DEVCOM Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）
Existing thermal-to-visible face verification approaches expect the thermal and visible face images to be of similar resolution. This is unlikely in real-world long-range surveillance systems, since humans are distant from the cameras. To address this issue, we introduce the task of thermal-to-visible face verification from low-resolution thermal images. Furthermore, we propose Axial-Generative Adversarial Network (Axial-GAN) to synthesize high-resolution visible images for matching. In the proposed approach we augment the GAN framework with axial-attention layers which leverage the recent adv…
Bringing Multimodal Large Language Models to Infrared-Visible Image Fusion Quality Assessment
Northwestern University（西北大学）；Northeastern University（东北大学）；University of Washington（华盛顿大学）；Hong Kong Baptist University（香港 Baptist大学）；Beijing Normal - Hong Kong Baptist University（北京师范大学-香港 Baptist大学）
Direction-aware multi-scale gradient loss for infrared and visible image fusion
Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences（光电信息处理重点实验室，中国科学院）；Shenyang Institute of Automation, Chinese Academy of Sciences（沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences, Chinese Academy of Sciences（中国科学院大学）
VOIC: Visible-Occluded Integrated Guidance for 3D Semantic Scene Completion
Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（可信自主系统研究院，南方科技大学）；Department of Computer Science and Engineering, Southern University of Science and Technology（计算机科学与工程系，南方科技大学）；School of Computer Science, University of Nottingham Ningbo China（宁波大学计算机学院）；Department of Electronic and Information Engineering, Changchun University（电子与信息工程学院，长春大学）
Perceptual Region-Driven Infrared-Visible Co-Fusion for Extreme Scene Enhancement
College of Aerospace Science and Engineering, National University of Defense Technology（航天科学与工程学院，国防科技大学）；Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation（湖南省图像测量与视觉导航重点实验室）；Beijing Institute of Tracking and Telecommunication Technology（北京跟踪与电信技术研究所）；National Key Laboratory of Space Integrated Information System（空间一体化信息系统国家重点实验室）
Frequency Routing and Identity Conditioning for Thermal-to-Visible Face Translation
Monash University, Monash Suzhou Research Institute（莫纳什大学、莫纳什大学苏州研究院）；Southeast University, School of Computer Science and Engineering（东南大学计算机科学与工程学院）；Monash University, Faculty of Information Technology（莫纳什大学信息技术学院）
Combined Dictionary Unfolding Network with Gradient-Adaptive Fidelity for Transferable Multi-Source Fusion
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）；College of Surveying and Geoinformatics, Tongji University（同济大学测绘与地理信息学院）；School of Computer and Information, Hefei University of Technology（合肥工业大学计算机与信息学院）
Integrated Laser Scanning and Image-Based Topology Optimization Techniques for Detection and Quantification of Visible and Subsurface Structural Defects
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）
Existing infrared and visible image fusion methods often face the dilemma of balancing modal information. Generative fusion methods reconstruct fused images by learning from data distributions, but their generative capabilities remain limited. Moreover, the lack of interpretability in modal information selection further affects the reliability and consistency of fusion results in complex scenarios. This manuscript revisits the essence of generative image fusion under the inspiration of human cognitive laws and proposes a novel infrared and visible image fusion method, termed HCLFuse. First, HC…
We propose POse-guided SElective Fusion (POSEFusion), a single-view human volumetric capture method that leverages tracking-based methods and tracking-free inference to achieve high-fidelity and dynamic 3D reconstruction. By contributing a novel reconstruction framework which contains pose-guided keyframe selection and robust implicit surface fusion, our method fully utilizes the advantages of both tracking-based methods and tracking-free inference methods, and finally enables the high-fidelity reconstruction of dynamic surface details even in the invisible regions. We formulate the keyframe s…
Rethinking the Evaluation of Visible and Infrared Image Fusion
School of Electronics and Information Engineering, Harbin Institute of Technology（哈尔滨工业大学电子与信息工程学院）；School of Computer and Communication Engineering, Northeastern University（东北大学计算机与通信工程学院）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）
$\textrm{A}^{\textrm{2}}$RNet: Adversarial Attack Resilient Network for Robust Infrared and Visible Image Fusion
School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；School of Software Technology, Dalian University of Technology（大连理工大学软件学院）
We present PHORHUM, a novel, end-to-end trainable, deep neural network methodology for photorealistic 3D human reconstruction given just a monocular RGB image. Our pixel-aligned method estimates detailed 3D geometry and, for the first time, the unshaded surface color together with the scene illumination. Observing that 3D supervision alone is not sufficient for high fidelity color reconstruction, we introduce patch-based rendering losses that enable reliable color reconstruction on visible parts of the human, and detailed and plausible color estimation for the non-visible parts. Moreover, our …
FlowObject: Flow Steering for Bridging Generative Priors and Reconstruction Fidelity
Graz University of Technology Austria（奥地利格拉茨理工大学）；Tampere University Finland（芬兰塔尔库大学）；Technical University of Munich Germany（德国慕尼黑技术大学）；Stanford University The United States of America（美国斯坦福大学）；Xi’an Jiaotong University China（中国西安交通大学）；École des Ponts ParisTech France（法国巴黎综合理工学院）
P2Fusion: Prompt-based Progressive Infrared-Visible Image Fusion via Dual-Prior Distillation
Northwestern Polytechnical University（西北工业大学）；Hefei University of Technology（合肥工业大学）；Rocket Force University of Engineering（火箭军工程大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Modern-day surveillance systems perform person recognition using deep learning-based face verification networks. Most state-of-the-art facial verification systems are trained using visible spectrum images. But, acquiring images in the visible spectrum is impractical in scenarios of low-light and nighttime conditions, and often images are captured in an alternate domain such as the thermal infrared domain. Facial verification in thermal images is often performed after retrieving the corresponding visible domain images. This is a well-established problem often known as the Thermal-to-Visible (T2…
To address the ill-posed problem caused by partial observations in monocular human volumetric capture, we present AvatarCap, a novel framework that introduces animatable avatars into the capture pipeline for high-fidelity reconstruction in both visible and invisible regions. Our method firstly creates an animatable avatar for the subject from a small number (~20) of 3D scans as a prior. Then given a monocular RGB video of this subject, our method integrates information from both the image observation and the avatar prior, and accordingly recon-structs high-fidelity 3D textured models with dyna…
Wuhan University（武汉大学）；North Carolina State University（北卡罗来纳州立大学）
This paper studies the problem of holistic 3D wireframe perception (HoW-3D), a new task of perceiving both the visible 3D wireframes and the invisible ones from single-view 2D images. As the non-front surfaces of an object cannot be directly observed in a single view, estimating the non-line-of-sight (NLOS) geometries in HoW-3D is a fundamentally challenging problem and remains open in computer vision. We study the problem of HoW-3D by proposing an ABC-HoW benchmark, which is created on top of CAD models sourced from the ABC-dataset with 12k single-view images and the corresponding holistic 3D…
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Visible-infrared person re-identification (VI-ReID) is an important task in night-time surveillance applications, since visible cameras are difficult to capture valid appearance information under poor illumination conditions. Compared to traditional person re-identification that handles only the intra-modality discrepancy, VI-ReID suffers from additional cross-modality discrepancy caused by different types of imaging systems. To reduce both intra- and cross-modality discrepancies, we propose a Hierarchical Cross-Modality Disentanglement (Hi-CMD) method, which automatically disentangles ID-disc…
Nanjing University of Science and Technology（南京理工大学）；University of Macau（澳门大学）
Cross-spectrum depth estimation aims to provide a depth map in all illumination conditions with a pair of dual-spectrum images. It is valuable for autonomous vehicle applications when the vehicle is equipped with two cameras of different modalities. However, images captured by different-modality cameras can be photometrically quite different. Therefore, cross-spectrum depth estimation is a very challenging problem. Moreover, the shortage of large-scale open-source datasets also retards further research in this field. In this paper, we propose an unsupervised visible-light image guided cross-sp…
SC-Diff: Semantically Calibrated Diffusion for Visible-to-Infrared Image Translation
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）
Guangzhou University（广州大学）；University of Macau（澳门大学）
Visible-infrared cross-modality person re-identification (VI-ReID), whose aim is to match person images between visible and infrared modality, is a challenging cross-modality image retrieval task. Most existing works integrate batch normalization layers into their neural network, but we found out that batch normalization layers would lead to two types of distribution gap: 1) inter-mini-batch distribution gap -- the distribution gap of the same modality between each mini-batch; 2) intra-mini-batch modality distribution gap -- the distribution gap of different modality within the same mini-batch…
Johns Hopkins University（约翰霍普金斯大学）；University of Nebraska（内布拉斯加大学）；U.S. Army DEVCOM Army Research Laboratory (ARL)（美国陆军DEVCOM陆军研究实验室）；Booz Allen Hamilton（博思艾伦汉密尔顿公司）
Thermal-to-visible face verification is a challenging problem due to the large domain discrepancy between the modalities. Existing approaches either attempt to synthesize visible faces from thermal faces or learn domain-invariant robust features from these modalities for cross-modal matching. In this paper, we use attributes extracted from visible images to synthesize attribute-preserved visible images from thermal imagery for cross-modal matching. A pre-trained attribute predictor network is used to extract the attributes from the visible image. Then, a novel multi-scale generator is proposed…
EUP Mataró Tecnocampus（马塔罗Tecnocampus高等理工学院）；Brno University of Technology（布尔诺理工大学）
In this paper we present a new database acquired with three different sensors (visible, near infrared and thermal) under different illumination conditions. This database consists of 41 people acquired in four different acquisition sessions, five images per session and three different illumination conditions. The total amount of pictures is 7.380 pictures. Experimental results are obtained through single sensor experiments as well as the combination of two and three sensors under different illumination conditions (natural, infrared and artificial illumination). We have found that the three spec…
