DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion
Xi’an Jiaotong University（西安交通大学）；Computer Vision Lab, ETH Zürich（苏黎世联邦理工学院计算机视觉实验室）；Northwestern Polytechnical University（西北工业大学）；Macau University of Science and Technology（澳门科技大学）；University of Würzburg（维尔茨堡大学）
Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, Luc Van Gool
Multi-modality image fusion aims to combine different modalities to produce fused images that retain the complementary features of each modality, such as functional highlights and texture details. To leverage strong generative priors and address challenges such as unstable training and lack of interpretability for GAN-based generative methods, we propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM). The fusion task is formulated as a conditional generation problem under the DDPM sampling framework, which is further divided into an unconditional generation subproblem and a maximum likelihood subproblem. The latter is modeled in a hierarchical Bayesian manner with latent variables and inferred by the expectation-maximization (EM) algorithm. By integrating the inference solution into the diffusion sampling iteration, our method can generate high-quality fused images with natural image generative priors and cross-modality information from source images. Note that all we required is an unconditional pre-trained generative model, and no fine-tuning is needed. Our extensive experiments indicate that our approach yields promising fusion results in infrared-visible image fusion and medical image fusion. The code is available at \url{https://github.com/Zhaozixiang1228/MMIF-DDFM}.
Enhancing Video Physical Consistency via Role-aware Joint Training and Modality-decoupled Denoising
University of Science and Technology of China（中国科学技术大学）；University of Electronic Science and Technology of China（电子科技大学）；Fudan University（复旦大学）；Georgia Institute of Technology（佐治亚理工学院）；Shanghai Jiao Tong University（上海交通大学）
Cycle-guided Denoising Diffusion Probability Model for 3D Cross-modality MRI Synthesis
Emory University（埃默里大学）；Winship Cancer Institute（温希普癌症研究所）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）
GARD: Gamma-based Anatomical Restoration and Denoising for Retinal OCT
Christian Doppler Laboratory for Artificial Intelligence in Retina, Center for Medical Data Science, Medical University of Vienna, 1090 Vienna, Austria（人工视网膜人工智能实验室，医学数据科学中心，维也纳医科大学）；Institute of Artificial Intelligence, Center for Medical Data Science, Medical University of Vienna, 1090 Vienna, Austria（人工智能研究所，医学数据科学中心，维也纳医科大学）
Adaptive Domain Shift in Diffusion Models for Cross-Modality Image Translation
Laplace Lab at University of Tennessee（田纳西大学Laplace实验室）；University of California Riverside（加州大学河滨分校）；University of Tennessee at Chattanooga（田纳西大学查塔努加分校）
Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Massachusetts Institute of Technology（麻省理工学院）；Tufts School of Medicine（塔夫茨医学院）
Optical coherence tomography angiography (OCTA) is a novel and clinically promising imaging modality to image retinal and sub-retinal vasculature. Based on repeated optical coherence tomography (OCT) scans, intensity changes are observed over time and used to compute OCTA image data. OCTA data are prone to noise and artifacts caused by variations in flow speed and patient movement. We propose a novel iterative maximum a posteriori signal recovery algorithm in order to generate OCTA volumes with reduced noise and increased image quality. This algorithm is based on previous work on probabilistic…
Centre Inria d’Université Côte d’Azur（蔚蓝海岸大学法国国家信息与自动化研究所中心）
Cross-modality data translation has attracted great interest in image computing. Deep generative models (\textit{e.g.}, GANs) show performance improvement in tackling those problems. Nevertheless, as a fundamental challenge in image translation, the problem of Zero-shot-Learning Cross-Modality Data Translation with fidelity remains unanswered. This paper proposes a new unsupervised zero-shot-learning method named Mutual Information guided Diffusion cross-modality data translation Model (MIDiffusion), which learns to translate the unseen source data to the target domain. The MIDiffusion leverag…
Volumetric Conditional Score-based Residual Diffusion Model for PET/MR Denoising
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Zhejiang University（浙江大学）；Tsinghua University（清华大学）；Neuroscience Research Institute（神经科学研究所）；Gachon University of Medicine and Science（嘉泉医科大学）
Nanjing University（南京大学）；SenseTime Research（商汤科技研究院）；CUHK - Sensetime Joint Lab（香港中文大学-商汤联合实验室）；Shanghai AI Laboratory（上海人工智能实验室）
This paper focuses on the weakly-supervised audio-visual video parsing task, which aims to recognize all events belonging to each modality and localize their temporal boundaries. This task is challenging because only overall labels indicating the video events are provided for training. However, an event might be labeled but not appear in one of the modalities, which results in a modality-specific noisy label problem. In this work, we propose a training strategy to identify and remove modality-specific noisy labels dynamically. It is motivated by two key observations: 1) networks tend to learn …
Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Process
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Westlake University（西交大学）；Zhejiang University（浙江大学）；Monash University（墨尔本大学）
In this paper, we propose a new multimodal image denoising approach to attenuate white Gaussian additive noise in a given image modality under the aid of a guidance image modality. The proposed coupled image denoising approach consists of two stages: coupled sparse coding and reconstruction. The first stage performs joint sparse transform for multimodal images with respect to a group of learned coupled dictionaries, followed by a shrinkage operation on the sparse representations. Then, in the second stage, the shrunken representations, together with coupled dictionaries, contribute to the reco…
Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification
The Guangdong Provincial Key Laboratory of Computational Intelligence and Cyberspace Information, the School of Computer Science and Engineering, South China University of Technology（广东省计算智能与网络信息重点实验室、计算机科学与工程学院、华南理工大学）；The Pazhou Laboratory（琶洲实验室）
MRI and CT are most widely used medical imaging modalities. It is often necessary to acquire multi-modality images for diagnosis and treatment such as radiotherapy planning. However, multi-modality imaging is not only costly but also introduces misalignment between MRI and CT images. To address this challenge, computational conversion is a viable approach between MRI and CT images, especially from MRI to CT images. In this paper, we propose to use an emerging deep learning framework called diffusion and score-matching models in this context. Specifically, we adapt denoising diffusion probabili…
Optical coherence tomography (OCT) is a prevalent imaging technique for retina. However, it is affected by multiplicative speckle noise that can degrade the visibility of essential anatomical structures, including blood vessels and tissue layers. Although averaging repeated B-scan frames can significantly improve the signal-to-noise-ratio (SNR), this requires longer acquisition time, which can introduce motion artifacts and cause discomfort to patients. In this study, we propose a learning-based method that exploits information from the single-frame noisy B-scan and a pseudo-modality that is c…
Sample-Aware Test-Time Adaptation for Medical Image-to-Image Translation
Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma（人工智能与计算机系统单位，工程系，罗马生物医学大学）；Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University（诊断与干预系，辐射物理，生物医学工程，乌梅大学）
Image denoising is getting more significance, especially in Computed Tomography (CT), which is an important and most common modality in medical imaging. This is mainly due to that the effectiveness of clinical diagnosis using CT image lies on the image quality. The denoising technique for CT images using window-based Multi-wavelet transformation and thresholding shows the effectiveness in denoising, however, a drawback exists in selecting the closer windows in the process of window-based multi-wavelet transformation and thresholding. Generally, the windows of the duplicate noisy image that are…
SUNY at Buffalo（纽约州立大学布法罗分校）；University of Electronic Science and Technology of China（电子科技大学）
Visual restoration and recognition are traditionally addressed in pipeline fashion, i.e. denoising followed by classification. Instead, observing correlations between the two tasks, for example clearer image will lead to better categorization and vice visa, we propose a joint framework for visual restoration and recognition for handwritten images, inspired by advances in deep autoencoder and multi-modality learning. Our model is a 3-pathway deep architecture with a hidden-layer representation which is shared by multi-inputs and outputs, and each branch can be composed of a multi-layer deep mod…
Unpaired image-to-image translation refers to learning inter-image-domain mapping in an unsupervised manner. Existing methods often learn deterministic mappings without explicitly modelling the robustness to outliers or predictive uncertainty, leading to performance degradation when encountering unseen out-of-distribution (OOD) patterns at test time. To address this limitation, we propose a novel probabilistic method called Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC), which models the per-pixel residual by generalized Gaussian distribution, capable of modelling heavy-tailed…
VS-DDPM: Efficient Low-Cost Diffusion Model for Medical Modality Translation
Center Algoritmi / LASI, University of Minho（中心Algoritmi / LASI，明霍大学）；Institute for Artificial Intelligence in Medicine (IKIM)（医学人工智能研究所）；Essen University Hospital (AöR)（埃森大学医院（AöR））；University of Duisburg-Essen（杜伊斯堡-埃森大学）；Cancer Research Center Cologne Essen (CCCE)（科隆-埃森癌症研究中心（CCCE））；West German Cancer Center（西德癌症中心）；University Hospital Essen (AöR)（埃森大学医院（AöR））；German Cancer Consortium (DKTK)（德国癌症联盟（DKTK））；Partner site University Hospital Essen (AöR)（埃森大学医院（AöR）合作站点）；Institute of Computer Graphics and Vision（计算机图形与视觉研究所）；Graz University of Technology（格拉茨技术大学）；Institute of Medical Informatics（医学信息学研究所）；University Hospital RWTH Aachen（亚琛RWTH大学医院）；Department of Oral and Maxillofacial Surgery（口腔和颌面外科部门）；TU Dortmund University（多特蒙德技术大学）；Center for Virtual and Extended Reality in Medicine (ZvRM)（医学虚拟和扩展现实中心（ZvRM））；Faculty of Computer Science（计算机科学学院）
As a powerful technique in medical imaging, image synthesis is widely used in applications such as denoising, super resolution and modality transformation etc. Recently, the revival of deep neural networks made immense progress in the field of medical imaging. Although many deep leaning based models have been proposed to improve the image synthesis accuracy, the evaluation of the model uncertainty, which is highly important for medical applications, has been a missing part. In this work, we propose to use Bayesian conditional generative adversarial network (GAN) with concrete dropout to improv…
Due to the limitations of the model structure and pre-training objectives, existing vision-and-language generation models cannot utilize pair-wise images and text through bi-directional generation. In this paper, we propose DU-VLG, a framework which unifies vision-and-language generation as sequence generation problems. DU-VLG is trained with novel dual pre-training tasks: multi-modal denoising autoencoder tasks and modality translation tasks. To bridge the gap between image understanding and generation, we further design a novel commitment loss. We compare pre-training objectives on image cap…
Cross-conditioned Diffusion Model for Medical Image to Image Translation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Xi’an Jiaotong University（西安交通大学）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
Diffusion models (DMs) have shown great potential for high-quality image synthesis. However, when it comes to producing images with complex scenes, how to properly describe both image global structures and object details remains a challenging task. In this paper, we present Frido, a Feature Pyramid Diffusion model performing a multi-scale coarse-to-fine denoising process for image synthesis. Our model decomposes an input image into scale-dependent vector quantized features, followed by a coarse-to-fine gating for producing image output. During the above multi-scale representation learning stag…
HEAL: Learning-Free Source Free Unsupervised Domain Adaptation for Cross-Modality Medical Image Segmentation
College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China（医学与生物信息工程学院，东北大学，沈阳，中国）；Key Laboratory of Medical Image Computing, Ministry of Education, Northeastern University, Shenyang, China（医学图像计算重点实验室，教育部，东北大学，沈阳，中国）；Engineering Research Center of Medical Imaging and Intelligent Analysis, Ministry of Education, Northeastern University, Shenyang, China（医学影像与智能分析工程研究中心，教育部，东北大学，沈阳，中国）
College of Physics and Information Engineering, Fuzhou University（福州大学物理与信息工程学院）；School of Basic Medical Science, Fudan University（复旦大学基础医学院）
Multi-sequence of cardiac magnetic resonance (CMR) images can provide complementary information for myocardial pathology (scar and edema). However, it is still challenging to fuse these underlying information for pathology segmentation effectively. This work presents an automatic cascade pathology segmentation framework based on multi-modality CMR images. It mainly consists of two neural networks: an anatomical structure segmentation network (ASSN) and a pathological region segmentation network (PRSN). Specifically, the ASSN aims to segment the anatomical structure where the pathology may exis…
Spatio-Temporal Mixture-of-Modality-Experts Diffusion for Quantitative DCE-MRI Synthesis from Incomplete MR Sequences
Interdisciplinary Program in Cancer Biology, Seoul National University College of Medicine（首尔大学医学院癌症生物学跨学科项目）；Department of Radiology, Seoul National University Hospital（首尔大学医院放射科）；Department of Radiology, Seoul National University College of Medicine（首尔大学医学院放射科）；Healthcare AI Research Institute, Seoul National University Hospital（首尔大学医院医疗人工智能研究所）
Optical Coherence Tomography (OCT) is pervasive in both the research and clinical practice of Ophthalmology. However, OCT images are strongly corrupted by noise, limiting their interpretation. Current OCT denoisers leverage assumptions on noise distributions or generate targets for training deep supervised denoisers via averaging of repeat acquisitions. However, recent self-supervised advances allow the training of deep denoising networks using only repeat acquisitions without clean targets as ground truth, reducing the burden of supervised learning. Despite the clear advantages of self-superv…
Self-Consistent Recursive Diffusion Bridge for Medical Image Translation
Bilkent University（毕尔肯大学）；National Magnetic Resonance Research Center (UMRAM)（国家磁共振研究中心）；Stanford University（斯坦福大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Modality-Guided Dynamic Graph Fusion and Temporal Diffusion for Self-Supervised RGB-T Tracking
School of Computer Sciences and Technology, China University of Mining and Technology（中国矿业大学计算机科学与技术学院）；Mine Digitization Engineering Research Center of the Ministry of Education, China（教育部 mine 数字化工程研究中心）
