Deep neural networks have been applied successfully to a wide variety of inverse problems arising in computational imaging. These networks are typically trained using a forward model that describes the measurement process to be inverted, which is often incorporated directly into the network itself. However, these approaches are sensitive to changes in the forward model: if at test time the forward model varies (even slightly) from the one the network was trained for, the reconstruction performance can degrade substantially. Given a network trained to solve an initial inverse problem with a known forward model, we propose two novel procedures that adapt the network to a change in the forward model, even without full knowledge of the change. Our approaches do not require access to more labeled data (i.e., ground truth images). We show these simple model adaptation approaches achieve empirical success in a variety of inverse problems, including deblurring, super-resolution, and undersampled image reconstruction in magnetic resonance imaging.
Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）
Photon-efficient imaging with the single-photon light detection and ranging (LiDAR) captures the three-dimensional (3D) structure of a scene by only a few detected signal photons per pixel. However, the existing computational methods for photon-efficient imaging are pre-tuned on a restricted scenario or trained on simulated datasets. When applied to realistic scenarios whose signal-to-background ratios (SBR) and other hardware-specific properties differ from those of the original task, the model performance often significantly deteriorates. In this paper, we present a domain adversarial adapta…
Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）
With the availability of commercial Light Field (LF) cameras, LF imaging has emerged as an up and coming technology in computational photography. However, the spatial resolution is significantly constrained in commercial microlens based LF cameras because of the inherent multiplexing of spatial and angular information. Therefore, it becomes the main bottleneck for other applications of light field cameras. This paper proposes an adaptation module in a pretrained Single Image Super Resolution (SISR) network to leverage the powerful SISR model instead of using highly engineered light field imagi…
Deep learning is emerging as a new paradigm for solving inverse imaging problems. However, the deep learning methods often lack the assurance of traditional physics-based methods due to the lack of physical information considerations in neural network training and deploying. The appropriate supervision and explicit calibration by the information of the physic model can enhance the neural network learning and its practical performance. In this paper, inspired by the geometry that data can be decomposed by two components from the null-space of the forward operator and the range space of its pseu…
In this survey paper, we review recent uses of convolution neural networks (CNNs) to solve inverse problems in imaging. It has recently become feasible to train deep CNNs on large databases of images, and they have shown outstanding performance on object classification and segmentation tasks. Motivated by these successes, researchers have begun to apply CNNs to the resolution of inverse problems such as denoising, deconvolution, super-resolution, and medical image reconstruction, and they have started to report improvements over state-of-the-art methods, including sparsity-based techniques suc…
Sun Yat-sen University（中山大学）；Tencent America（美国腾讯公司）
Due to the sophisticated imaging process, an identical scene captured by different cameras could exhibit distinct imaging patterns, introducing distinct proficiency among the super-resolution (SR) models trained on images from different devices. In this paper, we investigate a novel and practical task coded cross-device SR, which strives to adapt a real-world SR model trained on the paired images captured by one camera to low-resolution (LR) images captured by arbitrary target devices. The proposed task is highly challenging due to the absence of paired data from various imaging devices. To ad…
Machine learning methods for computational imaging require uncertainty estimation to be reliable in real settings. While Bayesian models offer a computationally tractable way of recovering uncertainty, they need large data volumes to be trained, which in imaging applications implicates prohibitively expensive collections with specific imaging instruments. This paper introduces a novel framework to train variational inference for inverse problems exploiting in combination few experimentally collected data, domain expertise and existing image data sets. In such a way, Bayesian machine learning m…
Conditional normalizing flows can generate diverse image samples for solving inverse problems. Most normalizing flows for inverse problems in imaging employ the conditional affine coupling layer that can generate diverse images quickly. However, unintended severe artifacts are occasionally observed in the output of them. In this work, we address this critical issue by investigating the origins of these artifacts and proposing the conditions to avoid them. First of all, we empirically and theoretically reveal that these problems are caused by "exploding inverse" in the conditional affine coupli…
DIPA: Distilled Preconditioned Algorithms for Solving Imaging Inverse Problems
Department of Computer Science, Universidad Industrial de Santander（圣安德烈斯工业大学计算机科学系）；Department of Electrical Engineering, Universidad Industrial de Santander（圣安德烈斯工业大学电气工程系）
Rice University（莱斯大学）；Indian Institute of Technology (IIT) Madras（印度理工学院马德拉斯分校）；Insight Centre for Data Analytics（数据 analytics 洞察中心）
Signal reconstruction is a challenging aspect of computational imaging as it often involves solving ill-posed inverse problems. Recently, deep feed-forward neural networks have led to state-of-the-art results in solving various inverse imaging problems. However, being task specific, these networks have to be learned for each inverse problem. On the other hand, a more flexible approach would be to learn a deep generative model once and then use it as a signal prior for solving various inverse problems. We show that among the various state of the art deep generative models, autoregressive models…
Inverse problems play a central role for many classical computer vision and image processing tasks. Many inverse problems are ill-posed, and hence require a prior to regularize the solution space. However, many of the existing priors, like total variation, are based on ad-hoc assumptions that have difficulties to represent the actual distribution of natural images. Thus, a key challenge in research on image processing is to find better suited priors to represent natural images. In this work, we propose the Adaptive Quantile Sparse Image (AQuaSI) prior. It is based on a quantile filter, can be …
University of Edinburgh（爱丁堡大学）；Noiseless Imaging Ltd（Noiseless成像有限公司）；University of Bath（巴斯大学）
In this work we investigate the practicality of stochastic gradient descent and recently introduced variants with variance-reduction techniques in imaging inverse problems. Such algorithms have been shown in the machine learning literature to have optimal complexities in theory, and provide great improvement empirically over the deterministic gradient methods. Surprisingly, in some tasks such as image deblurring, many of such methods fail to converge faster than the accelerated deterministic gradient methods, even in terms of epoch counts. We investigate this phenomenon and propose a theory-in…
Deep neural networks as image priors have been recently introduced for problems such as denoising, super-resolution and inpainting with promising performance gains over hand-crafted image priors such as sparsity and low-rank. Unlike learned generative priors they do not require any training over large datasets. However, few theoretical guarantees exist in the scope of using untrained neural network priors for inverse imaging problems. We explore new applications and theory for untrained neural network priors. Specifically, we consider the problem of solving linear inverse problems, such as com…
NF-ULA: Langevin Monte Carlo with Normalizing Flow Prior for Imaging Inverse Problems
Shanghai Jiao Tong University（上海交通大学）；University of Birmingham（伯明翰大学）；Indian Institute of Technology (IIT) Kharagpur（印度理工学院卡拉格布尔分校）；University of Cambridge（剑桥大学）
This work provides a strong baseline for the problem of multi-source multi-target domain adaptation and generalization in medical imaging. Using a diverse collection of ten chest X-ray datasets, we empirically demonstrate the benefits of training medical imaging deep learning models on varied patient populations for generalization to out-of-sample domains.
Philips Research（飞利浦研究院）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）
Generalizability of deep learning models may be severely affected by the difference in the distributions of the train (source domain) and the test (target domain) sets, e.g., when the sets are produced by different hardware. As a consequence of this domain shift, a certain model might perform well on data from one clinic, and then fail when deployed in another. We propose a very light and transparent approach to perform test-time domain adaptation. The idea is to substitute the target low-frequency Fourier space components that are deemed to reflect the style of an image. To maximize the perfo…
Zero-Shot Solving of Imaging Inverse Problems via Noise-Refined Likelihood Guided Diffusion Models
School of Mathematics and Statistics（数学与统计学学院）；Nanjing University of Science and Technology（南京理工大学）；School of Computer Science and Engineering（计算机科学与工程学院）
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Marquette University（马凯特大学）；University of Chicago（芝加哥大学）
Recent efforts on solving inverse problems in imaging via deep neural networks use architectures inspired by a fixed number of iterations of an optimization method. The number of iterations is typically quite small due to difficulties in training networks corresponding to more iterations; the resulting solvers cannot be run for more iterations at test time without incurring significant errors. This paper describes an alternative approach corresponding to an infinite number of iterations, yielding a consistent improvement in reconstruction accuracy above state-of-the-art alternatives and where …
In the past few years, deep learning-based methods have demonstrated enormous success for solving inverse problems in medical imaging. In this work, we address the following question:\textit{Given a set of measurements obtained from real imaging experiments, what is the best way to use a learnable model and the physics of the modality to solve the inverse problem and reconstruct the latent image?} Standard supervised learning based methods approach this problem by collecting data sets of known latent images and their corresponding measurements. However, these methods are often impractical due …
UCL（伦敦大学学院）；UCLH NHS Foundation Trust（伦敦大学学院医院国民保健服务基金会信托）
Any novel medical imaging modality that differs from previous protocols e.g. in the number of imaging channels, introduces a new domain that is heterogeneous from previous ones. This common medical imaging scenario is rarely considered in the domain adaptation literature, which handles shifts across domains of the same dimensionality. In our work we rely on stochastic generative modeling to translate across two heterogeneous domains at pixel space and introduce two new loss functions that promote semantic consistency. Firstly, we introduce a semantic cycle-consistency loss in the source domain…
University of Pennsylvania（宾夕法尼亚大学）；Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）
Domain shift, the mismatch between training and testing data characteristics, causes significant degradation in the predictive performance in multi-source imaging scenarios. In medical imaging, the heterogeneity of population, scanners and acquisition protocols at different sites presents a significant domain shift challenge and has limited the widespread clinical adoption of machine learning models. Harmonization methods which aim to learn a representation of data invariant to these differences are the prevalent tools to address domain shift, but they typically result in degradation of predic…
GMN4AD: Graph Matching Network for Alzheimer's Disease Diagnosis with Test-Time Domain Adaptation using Multi-centered Structure Magnetic Resonance Imaging
Department of Computer Science, Kennesaw State University（肯纳邦大学计算机科学系）；Department of Information Technology, Kennesaw State University（肯纳邦大学信息技术系）；School of Data Science and Analytics, Kennesaw State University（肯纳邦大学数据科学与分析学院）；Department of Applied Computing, Michigan Technological University（密歇根技术大学应用计算系）
University of Oxford（牛津大学）；NIHR Oxford Biomedical Research Centre（NIHR牛津生物医学研究中心）；Oxford University Hospitals NHS Foundation Trust（牛津大学医院NHS基金会信托）；John Radcliffe Hospital（约翰·拉德克利夫医院）
Barrett's oesophagus (BE) is one of the early indicators of esophageal cancer. Patients with BE are monitored and undergo ablation therapies to minimise the risk, thereby making it eminent to identify the BE area precisely. Automated segmentation can help clinical endoscopists to assess and treat BE area more accurately. Endoscopy imaging of BE can include multiple modalities in addition to the conventional white light (WL) modality. Supervised models require large amount of manual annotations incorporating all data variability in the training data. However, it becomes cumbersome, tedious and …
Ill-posed inverse problems in imaging remain an active research topic in several decades, with new approaches constantly emerging. Recognizing that the popular dictionary learning and convolutional sparse coding are both essentially modeling the high-frequency component of an image, which convey most of the semantic information such as texture details, in this work we propose a novel multi-profile high-frequency transform-guided denoising autoencoder as prior (HF-DAEP). To achieve this goal, we first extract a set of multi-profile high-frequency components via a specific transformation and add…
Synthetic aperture radar (SAR) imagery can provide useful information in a multitude of applications, including climate change, environmental monitoring, meteorology, high dimensional mapping, ship monitoring, or planetary exploration. In this paper, we investigate solutions to a number of inverse problems encountered in SAR imaging. We propose a convex proximal splitting method for the optimization of a cost function that includes a non-convex Cauchy-based penalty. The convergence of the overall cost function optimization is ensured through careful selection of model parameters within a forwa…
There has been significant recent interest in the use of deep learning for regularizing imaging inverse problems. Most work in the area has focused on regularization imposed implicitly by convolutional neural networks (CNNs) pre-trained for image reconstruction. In this work, we follow an alternative line of work based on learning explicit regularization functionals that promote preferred solutions. We develop the Explicit Learned Deep Equilibrium Regularizer (ELDER) method for learning explicit regularizers that minimize a mean-squared error (MSE) metric. ELDER is based on a regularization fu…
Reconstructing medical images from partial measurements is an important inverse problem in Computed Tomography (CT) and Magnetic Resonance Imaging (MRI). Existing solutions based on machine learning typically train a model to directly map measurements to medical images, leveraging a training dataset of paired images and measurements. These measurements are typically synthesized from images using a fixed physical model of the measurement process, which hinders the generalization capability of models to unknown measurement processes. To address this issue, we propose a fully unsupervised techniq…
Imaging through scattering media is a useful and yet demanding task since it involves solving for an inverse mapping from speckle images to object images. It becomes even more challenging when the scattering medium undergoes dynamic changes. Various approaches have been proposed in recent years. However, to date, none is able to preserve high image quality without either assuming a finite number of sources for dynamic changes, assuming a thin scattering medium, or requiring the access to both ends of the medium. In this paper, we propose an adaptive inverse mapping (AIP) method which is flexib…
Recently, with the significant developments in deep learning techniques, solving underdetermined inverse problems has become one of the major concerns in the medical imaging domain. Typical examples include undersampled magnetic resonance imaging, interior tomography, and sparse-view computed tomography, where deep learning techniques have achieved excellent performances. Although deep learning methods appear to overcome the limitations of existing mathematical methods when handling various underdetermined problems, there is a lack of rigorous mathematical foundations that would allow us to el…
Institute of Sensors, Signals；Systems, \,Heriot-Watt University,\, Edinburgh,\, UK
The remarkable performance of deep neural networks (DNNs) currently makes them the method of choice for solving linear inverse problems. They have been applied to super-resolve and restore images, as well as to reconstruct MR and CT images. In these applications, DNNs invert a forward operator by finding, via training data, a map between the measurements and the input images. It is then expected that the map is still valid for the test data. This framework, however, introduces measurement inconsistency during testing. We show that such inconsistency, which can be critical in domains like medic…
Computational imaging systems jointly design computation and hardware to retrieve information which is not traditionally accessible with standard imaging systems. Recently, critical aspects such as experimental design and image priors are optimized through deep neural networks formed by the unrolled iterations of classical physics-based reconstructions (termed physics-based networks). However, for real-world large-scale systems, computing gradients via backpropagation restricts learning due to memory limitations of graphical processing units. In this work, we propose a memory-efficient learnin…
Multidimensional imaging, capturing image data in more than two dimensions, has been an emerging field with diverse applications. Due to the limitation of two-dimensional detectors in obtaining the high-dimensional image data, computational imaging approaches have been developed to pass on some of the burden to a reconstruction algorithm. In various image reconstruction problems in multidimensional imaging, the measurements are in the form of superimposed convolutions. In this paper, we introduce a general framework for the solution of these problems, called here convolutional inverse problems…
Hamburg University of Technology（汉堡工业大学）；Heidelberg University Hospital（海德堡大学医院）
We present Posterior Temperature Optimized Bayesian Inverse Models (POTOBIM), an unsupervised Bayesian approach to inverse problems in medical imaging using mean-field variational inference with a fully tempered posterior. Bayesian methods exhibit useful properties for approaching inverse tasks, such as tomographic reconstruction or image denoising. A suitable prior distribution introduces regularization, which is needed to solve the ill-posed problem and reduces overfitting the data. In practice, however, this often results in a suboptimal posterior temperature, and the full potential of the …
By co-designing a meta-optical front end in conjunction with an image-processing back end, we demonstrate noise sensitivity and compactness substantially superior to either an optics-only or a computation-only approach, illustrated by two examples: subwavelength imaging and reconstruction of the full polarization coherence matrices of multiple light sources. Our end-to-end inverse designs couple the solution of the full Maxwell equations---exploiting all aspects of wave physics arising in subwavelength scatterers---with inverse-scattering algorithms in a single large-scale optimization involvi…
OFDM-Based ISAC Imaging of Extended Targets via Inverse Virtual Aperture Processing
Dept. of Electrical, Electronic, and Inf. Eng. “Guglielmo Marconi,” University of Bologna（博洛尼亚大学吉列尔莫·马可尼电气、电子与信息工程学院）；CNIT/WiLab, Italy（意大利国家信息技术研究中心/WiLab）
Rice University（莱斯大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Chicago（芝加哥大学）；Stanford University（斯坦福大学）
Recent work in machine learning shows that deep neural networks can be used to solve a wide variety of inverse problems arising in computational imaging. We explore the central prevailing themes of this emerging area and present a taxonomy that can be used to categorize different problems and reconstruction methods. Our taxonomy is organized along two central axes: (1) whether or not a forward model is known and to what extent it is used in training and testing, and (2) whether or not the learning is supervised or unsupervised, i.e., whether or not the training relies on access to matched grou…
