University of Cambridge（剑桥大学）；CNRS（法国国家科学研究中心）；Université Côte d’Azur（蔚蓝海岸大学）；INRIA（法国国家信息与自动化研究所）；Univ. Bordeaux（波尔多大学）；Bordeaux INP（波尔多国立理工学院）；LaBRI（波尔多计算机科学研究所）；IMB（波尔多数学研究所）
Simone Parisotto, Luca Calatroni, Aurélie Bugeau, Nicolas Papadakis, Carola-Bibiane Schönlieb
We propose a new variational model for non-linear image fusion. Our approach is based on the use of an osmosis energy term related to the one studied in Vogel et al. (2013) and Weickert et al. (2013) The minimization of the proposed non-convex energy realizes visually plausible image data fusion, invariant to multiplicative brightness changes. On the practical side, it requires minimal supervision and parameter tuning and can encode prior information on the structure of the images to be fused. For the numerical solution of the proposed model, we develop a primal-dual algorithm and we apply the resulting minimization scheme to solve multi-modal face fusion, color transfer and cultural heritage conservation problems. Visual and quantitative comparisons to state-of-the-art approaches prove the out-performance and the flexibility of our method.
Bosch Center for Artificial Intelligence（博世人工智能中心）；University of Tübingen（图宾根大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
Sensor fusion can significantly improve the performance of many computer vision tasks. However, traditional fusion approaches are either not data-driven and cannot exploit prior knowledge nor find regularities in a given dataset or they are restricted to a single application. We overcome this shortcoming by presenting a novel deep hierarchical variational autoencoder called FusionVAE that can serve as a basis for many fusion tasks. Our approach is able to generate diverse image samples that are conditioned on multiple noisy, occluded, or only partially visible input images. We derive and optim…
Chandigarh Engineering College（昌迪加尔工程学院）；Thapar Institute of Engineering & Technology, University（塔帕尔工程技术大学）
Recent innovations shows that blending of details captured by single Low Dynamic Range (LDR) sensor overcomes the limitations of standard digital cameras to capture details from high dynamic range scene. We present a method to produce well-exposed fused image that can be displayed directly on conventional display devices. The ambition is to preserve details in poorly illuminated and brightly illuminated regions. Proposed approach does not require true radiance reconstruction and tone manipulation steps. The aforesaid objective is achieved by taking into account local information measure that s…
We introduce a novel model for spatially varying variational data fusion, driven by point-wise confidence values. The proposed model allows for the joint estimation of the data and the confidence values based on the spatial coherence of the data. We discuss the main properties of the introduced model as well as suitable algorithms for estimating the solution of the corresponding biconvex minimization problem and their convergence. The performance of the proposed model is evaluated considering the problem of depth image fusion by using both synthetic and real data from publicly available datase…
Hyperspectral imaging is a cutting-edge type of remote sensing used for mapping vegetation properties, rock minerals and other materials. A major drawback of hyperspectral imaging devices is their intrinsic low spatial resolution. In this paper, we propose a method for increasing the spatial resolution of a hyperspectral image by fusing it with an image of higher spatial resolution that was obtained with a different imaging modality. This is accomplished by solving a variational problem in which the regularization functional is the directional total variation. To accommodate for possible mis-r…
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）
Existing infrared and visible image fusion methods often face the dilemma of balancing modal information. Generative fusion methods reconstruct fused images by learning from data distributions, but their generative capabilities remain limited. Moreover, the lack of interpretability in modal information selection further affects the reliability and consistency of fusion results in complex scenarios. This manuscript revisits the essence of generative image fusion under the inspiration of human cognitive laws and proposes a novel infrared and visible image fusion method, termed HCLFuse. First, HC…
Bayesian Fully-Connected Tensor Network for Hyperspectral-Multispectral Image Fusion
School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Computer Science and Artificial Intelligence, Zhengzhou University（郑州大学计算机科学与人工智能学院）；Department of Computer Science, St. Francis Xavier University（圣弗朗西斯大学计算机科学系）
Volume-based reconstruction is usually expensive both in terms of memory consumption and runtime. Especially for sparse geometric structures, volumetric representations produce a huge computational overhead. We present an efficient way to fuse range data via a variational Octree-based minimization approach by taking the actual range data geometry into account. We transform the data into Octree-based truncated signed distance fields and show how the optimization can be conducted on the newly created structures. The main challenge is to uphold speed and a low memory footprint without sacrificing…
Imperial College London（帝国理工学院）；University of Twente（特文特大学）
We address the task of simultaneous feature fusion and modeling of discrete ordinal outputs. We propose a novel Gaussian process(GP) auto-encoder modeling approach. In particular, we introduce GP encoders to project multiple observed features onto a latent space, while GP decoders are responsible for reconstructing the original features. Inference is performed in a novel variational framework, where the recovered latent representations are further constrained by the ordinal output labels. In this way, we seamlessly integrate the ordinal structure in the learned manifold, while attaining robust…
Unleashing Vision Foundation Models for Coronary Artery Segmentation: Parallel ViT-CNN Encoding and Variational Fusion
National-Local Joint Engineering Research Center of Biodiagnosis \& Biotherapy, the Second Affiliated Hospital of Xi’an Jiaotong University, Xi’an, 710004, China Institute of Medical Artificial Intelligence, the Second Affiliated Hospital of Xi’an Jiaotong University, Xi'an, 710004, China Viadrina European University, Frankfurt (Oder), 15230, Germany
National University of Water and Environmental Engineering（国立水利与环境工程大学）；EOS Data Analytics Ukraine（乌克兰EOS数据分析公司）；Oles Honchar Dnipro National University（奥列西·贡恰尔第聂伯国立大学）
In this paper we discuss a new variational approach to the Date Fusion problem of multi-spectral satellite images from Sentinel-2 and MODIS that have been captured at different resolution level and, arguably, on different days. The crucial point of our approach that the MODIS image is cloud-free whereas the images from Sentinel-2 can be corrupted by clouds or noise.
Most satellites decouple the acquisition of a panchromatic image at high spatial resolution from the acquisition of a multispectral image at lower spatial resolution. Pansharpening is a fusion technique used to increase the spatial resolution of the multispectral data while simultaneously preserving its spectral information. In this paper, we consider pansharpening as an optimization problem minimizing a cost function with a nonlocal regularization term. The energy functional which is to be minimized decouples for each band, thus permitting the application to misregistered spectral components.…
In this paper we address the problem of tracking multiple speakers via the fusion of visual and auditory information. We propose to exploit the complementary nature of these two modalities in order to accurately estimate smooth trajectories of the tracked persons, to deal with the partial or total absence of one of the modalities over short periods of time, and to estimate the acoustic status -- either speaking or silent -- of each tracked person along time. We propose to cast the problem at hand into a generative audio-visual fusion (or association) model formulated as a latent-variable tempo…
Technical University of Munich（慕尼黑工业大学）；University of Toulouse（图卢兹大学）；Mikros Image
We introduce a variational method for multi-view shape-from-shading under natural illumination. The key idea is to couple PDE-based solutions for single-image based shape-from-shading problems across multiple images and multiple color channels by means of a variational formulation. Rather than alternatingly solving the individual SFS problems and optimizing the consistency across images and channels which is known to lead to suboptimal results, we propose an efficient solution of the coupled problem by means of an ADMM algorithm. In numerous experiments on both simulated and real imagery, we d…
Department of Electrical Engineering and Automation, Aalto University, Finland（艾尔沃斯大学电子工程与自动化系）；Department of Computer Science, Aalto University, Finland（艾尔沃斯大学计算机科学系）；Center for Machine Vision and Signal Analysis, University of Oulu, Finland（奥卢大学机器视觉与信号分析中心）
Turbulence-degraded image frames are distorted by both turbulent deformations and space-time-varying blurs. To suppress these effects, we propose a multi-frame reconstruction scheme to recover a latent image from the observed image sequence. Recent approaches are commonly based on registering each frame to a reference image, by which geometric turbulent deformations can be estimated and a sharp image can be restored. A major challenge is that a fine reference image is usually unavailable, as every turbulence-degraded frame is distorted. A high-quality reference image is crucial for the accurat…
Attention-Based Variational Framework for Joint and Individual Components Learning with Applications in Brain Network Analysis
Department of Biostatistics, Yale School of Public Health（生物统计学系，耶鲁公共卫生学院）；Department of Statistics, Virginia Tech（统计学系，弗吉尼亚理工大学）；Department of Statistics and Operations Research, University of North Carolina at Chapel Hill（统计学与运筹学系，北卡罗来纳大学教堂山分校）
SRUG: A Fusion-Driven Generator Network for Medical Image Translation
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science（上海工程技术大学电子电气工程学院）；Department of Thoracic Surgery, Zhongshan Hospital of Fudan University（复旦大学中山医院胸外科）；Clinical Research Unit, Institute of Clinical Science, Zhongshan Hospital of Fudan University（复旦大学中山医院临床科研部）；Department of Medical Oncology, Cancer Center, and Fudan Zhangjiang Institute, Zhongshan Hospital of Fudan University（复旦大学中山医院医学肿瘤科、癌症中心及复旦张江研究院）
