Model-based learned iterative reconstruction methods have recently been shown to outperform classical reconstruction algorithms. Applicability of these methods to large scale inverse problems is however limited by the available memory for training and extensive training times, the latter due to computationally expensive forward models. As a possible solution to these restrictions we propose a multi-scale learned iterative reconstruction scheme that computes iterates on discretisations of increasing resolution. This procedure does not only reduce memory requirements, it also considerably speeds up reconstruction and training times, but most importantly is scalable to large scale inverse problems with non-trivial forward operators, such as those that arise in many 3D tomographic applications. In particular, we propose a hybrid network that combines the multi-scale iterative approach with a particularly expressive network architecture which in combination exhibits excellent scalability in 3D.
Applicability of the algorithm is demonstrated for 3D cone beam computed tomography from real measurement data of an organic phantom. Additionally, we examine scalability and reconstruction quality in comparison to established learned reconstruction methods in two dimensions for low dose computed tomography on human phantoms.
Learned iterative networks: An operator learning perspective
Research Unit of Mathematical Sciences, University of Oulu（奥卢大学数学科学研究所）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）；Department of Mathematics, KTH - Royal Institute of Technology（瑞典皇家理工学院数学系）
We introduce the learned simultaneous iterative reconstruction technique (SIRT) for tomographic reconstruction. The learned SIRT algorithm is a deep learning based reconstruction method combining model knowledge with a learned component. The algorithm is trained by mapping raw measured data to the reconstruction results over several iterations. The Learned SIRT algorithm is applied to a cone beam geometry on a circular orbit, a challenging problem for learned methods due to its 3D geometry and its inherent inability to completely capture the patient anatomy. A comparison of 2D reconstructions …
University of Florida（佛罗里达大学）；Sichuan University（四川大学）；Georgia State University（佐治亚州立大学）
We propose a provably convergent method, called Efficient Learned Descent Algorithm (ELDA), for low-dose CT (LDCT) reconstruction. ELDA is a highly interpretable neural network architecture with learned parameters and meanwhile retains convergence guarantee as classical optimization algorithms. To improve reconstruction quality, the proposed ELDA also employs a new non-local feature mapping and an associated regularizer. We compare ELDA with several state-of-the-art deep image methods, such as RED-CNN and Learned Primal-Dual, on a set of LDCT reconstruction problems. Numerical experiments demo…
Dose reduction in computed tomography (CT) is essential for decreasing radiation risk in clinical applications. Iterative reconstruction is one of the most promising ways to compensate for the increased noise due to reduction of photon flux. Rather than most existing prior-driven algorithms that benefit from manually designed prior functions or supervised learning schemes, in this work we integrate the data-consistency as a conditional term into the iterative generative model for low-dose CT. At the stage of prior learning, the gradient of data density is directly learned from normal-dose CT i…
In this work, we investigate the application of deep learning methods for computed tomography in the context of having a low-data regime. As motivation, we review some of the existing approaches and obtain quantitative results after training them with different amounts of data. We find that the learned primal-dual has an outstanding performance in terms of reconstruction quality and data efficiency. However, in general, end-to-end learned methods have two issues: a) lack of classical guarantees in inverse problems and b) lack of generalization when not trained with enough data. To overcome the…
It's well-known that inverse problems are ill-posed and to solve them meaningfully one has to employ regularization methods. Traditionally, popular regularization methods have been the penalized Variational approaches. In recent years, the classical regularized-reconstruction approaches have been outclassed by the (deep-learning-based) learned reconstruction algorithms. However, unlike the traditional regularization methods, the theoretical underpinnings, such as stability and regularization, have been insufficient for such learned reconstruction algorithms. Hence, the results obtained from su…
Coherent diffractive imaging (CDI), using both X-rays and electrons, has made extremely rapid progress over the past two decades. The associated reconstruction algorithms are typically iterative, and seeded with a crude first estimate. A deterministic method for Bragg Coherent Diffraction Imaging (Pavlov et al., Sci. Rep. 7, 1132 (2017)) is used as a more refined starting point for a shrink-wrap iterative reconstruction procedure. The appropriate comparison with the autocorrelation function as a starting point is performed. Real-space and Fourier-space error metrics are used to analyse the con…
When dealing with material classification in baggage at airports, Dual-Energy Computed Tomography (DECT) allows characterization of any given material with coefficients based on two attenuative effects: Compton scattering and photoelectric absorption. However, straightforward projection-domain decomposition methods for this characterization often yield poor reconstructions due to the high dynamic range of material properties encountered in an actual luggage scan. Hence, for better reconstruction quality under a timing constraint, we propose a splitting-based, GPU-accelerated, statistical DECT …
University of Würzburg（维尔茨堡大学）；Fraunhofer Institute for Integrated Circuits IIS（弗劳恩霍夫集成电路研究所IIS）；University Hospital Würzburg（维尔茨堡大学医院）；Research Centre for Natural Sciences（自然科学研究中心）
Purpose: To evaluate an iterative learning approach for enhanced performance of Robust Artificial-neural-networks for K-space Interpolation (RAKI), when only a limited amount of training data (auto-calibration signals, ACS) are available for accelerated standard 2D imaging. Methods: In a first step, the RAKI model was optimized for the case of strongly limited training data amount. In the iterative learning approach (termed iterative RAKI), the optimized RAKI model is initially trained using original and augmented ACS obtained from a linear parallel imaging reconstruction. Subsequently, the RA…
University College London（伦敦大学学院）；University of Oulu（奥卢大学）；The Chinese University of Hong Kong（香港中文大学）
Deep learning-based image reconstruction approaches have demonstrated impressive empirical performance in many imaging modalities. These approaches usually require a large amount of high-quality paired training data, which is often not available in medical imaging. To circumvent this issue we develop a novel unsupervised knowledge-transfer paradigm for learned reconstruction within a Bayesian framework. The proposed approach learns a reconstruction network in two phases. The first phase trains a reconstruction network with a set of ordered pairs comprising of ground truth images of ellipses an…
Friedrich-Alexander Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
Learned iterative reconstruction algorithms for inverse problems offer the flexibility to combine analytical knowledge about the problem with modules learned from data. This way, they achieve high reconstruction performance while ensuring consistency with the measured data. In computed tomography, extending such approaches from 2D fan-beam to 3D cone-beam data is challenging due to the prohibitively high GPU memory that would be needed to train such models. This paper proposes to use neural ordinary differential equations to solve the reconstruction problem in a residual formulation via numeri…
Compared to standard tomographic reconstruction, iterative approaches offer the possibility to account for extraneous experimental influences, which allows for a suppression of related artifacts. However, the inclusion of corresponding parameters in the iterative forward model typically leads to longer computation times. Here, we demonstrate experimentally for phase sensitive X-ray imaging based on the edge illumination principle that inadequately sampled illumination curves result in ring artifacts in tomographic reconstructions. We take advantage of appropriately sampled illumination curves …
Rutgers University（罗格斯大学）；New York University（纽约大学）
Magnetic Resonance (MR) image reconstruction from highly undersampled $k$-space data is critical in accelerated MR imaging (MRI) techniques. In recent years, deep learning-based methods have shown great potential in this task. This paper proposes a learned half-quadratic splitting algorithm for MR image reconstruction and implements the algorithm in an unrolled deep learning network architecture. We compare the performance of our proposed method on a public cardiac MR dataset against DC-CNN and LPDNet, and our method outperforms other methods in both quantitative results and qualitative result…
Cornell University（康奈尔大学）；Weill Medical College of Cornell University（康奈尔大学威尔医学院）
Quantitative susceptibility mapping (QSM) involves acquisition and reconstruction of a series of images at multi-echo time points to estimate tissue field, which prolongs scan time and requires specific reconstruction technique. In this paper, we present our new framework, called Learned Acquisition and Reconstruction Optimization (LARO), which aims to accelerate the multi-echo gradient echo (mGRE) pulse sequence for QSM. Our approach involves optimizing a Cartesian multi-echo k-space sampling pattern with a deep reconstruction network. Next, this optimized sampling pattern was implemented in …
Low Dose Computed Tomography suffers from a high amount of noise and/or undersampling artefacts in the reconstructed image. In the current article, a Deep Learning technique is exploited as a regularization term for the iterative reconstruction method SIRT. While SIRT minimizes the error in the sinogram space, the proposed regularization model additionally steers intermediate SIRT reconstructions towards the desired output. Extensive evaluations demonstrate the superior outcomes of the proposed method compared to the state of the art techniques. Comparing the forward projection of the reconstr…
University of Cambridge（剑桥大学）；KTH – Royal Institute of Technology（瑞典皇家理工学院）
In numerous practical applications, especially in medical image reconstruction, it is often infeasible to obtain a large ensemble of ground-truth/measurement pairs for supervised learning. Therefore, it is imperative to develop unsupervised learning protocols that are competitive with supervised approaches in performance. Motivated by the maximum-likelihood principle, we propose an unsupervised learning framework for solving ill-posed inverse problems. Instead of seeking pixel-wise proximity between the reconstructed and the ground-truth images, the proposed approach learns an iterative recons…
Tomography has made a revolutionary impact on diverse fields, ranging from macro-/mesoscopic scale studies in biology, radiology, plasma physics to the characterization of 3D atomic structure in material science. The fundamental of tomography is to reconstruct a 3D object from a set of 2D projections. To solve the tomography problem, many algorithms have been developed. Among them are methods using transformation technique such as computed tomography (CT) based on Radon transform and Generalized Fourier iterative reconstruction (GENFIRE) based on Fourier slice theorem (FST), and direct methods…
University of Leeds（利兹大学）；University of Milan-Bicocca（米兰比可卡大学）
Reconstructing images from downsampled and noisy measurements, such as MRI and low dose Computed Tomography (CT), is a mathematically ill-posed inverse problem. We propose an easy-to-use reconstruction method based on Expectation Propagation (EP) techniques. We incorporate the Monte Carlo (MC) method, Markov Chain Monte Carlo (MCMC), and Alternating Direction Method of Multiplier (ADMM) algorithm into EP method to address the intractability issue encountered in EP. We demonstrate the approach on complex Bayesian models for image reconstruction. Our technique is applied to images from Gamma-cam…
Enabling self-supervised learned primal dual with Noise2Inverse
Research Unit of Mathematical Sciences, University of Oulu, Finland（奥卢大学数学科学研究中心，芬兰）；Department of Mathematics and Information Science, Josai University, Japan（立命馆大学数学与信息科学系，日本）；Department of Computer Science, University College London, United Kingdom（伦敦大学学院计算机科学系，英国）
Graz University of Technology（格拉茨技术大学）；Silicon Austria Labs（奥地利硅谷实验室）；University of Bonn（波恩大学）；Johannes Kepler University（约翰·开普勒大学）；Technical University of Munich（慕尼黑工业大学）；Imperial College London（伦敦帝国理工学院）；NYU School of Medicine（纽约大学医学院）
Recent deep learning approaches focus on improving quantitative scores of dedicated benchmarks, and therefore only reduce the observation-related (aleatoric) uncertainty. However, the model-immanent (epistemic) uncertainty is less frequently systematically analyzed. In this work, we introduce a Bayesian variational framework to quantify the epistemic uncertainty. To this end, we solve the linear inverse problem of undersampled MRI reconstruction in a variational setting. The associated energy functional is composed of a data fidelity term and the total deep variation (TDV) as a learned paramet…
Zhejiang University（浙江大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Florida（佛罗里达大学）；Massachusetts General Brigham（麻省总医院布莱根医疗系统）
Deep learning based PET image reconstruction methods have achieved promising results recently. However, most of these methods follow a supervised learning paradigm, which rely heavily on the availability of high-quality training labels. In particular, the long scanning time required and high radiation exposure associated with PET scans make obtaining this labels impractical. In this paper, we propose a dual-domain unsupervised PET image reconstruction method based on learned decent algorithm, which reconstructs high-quality PET images from sinograms without the need for image labels. Specifica…
For single source helical Computed Tomography (CT), both Filtered-Back Projection (FBP) and statistical iterative reconstruction have been investigated. However for dual source CT with flying focal spot (DS-FFS CT), statistical iterative reconstruction that accurately models the scanner geometry and physics remains unknown to researchers. Therefore, this paper presents a novel physics-based iterative reconstruction method for DS-FFS CT and assess its image quality. Our algorithm uses precise physics models to reconstruct from the native cone-beam geometry and interleaved dual source helical tr…
Resting-state functional Magnetic Resonance Imaging (fMRI) is a powerful imaging technique for studying functional development of the brain in utero. However, unpredictable and excessive movement of fetuses have limited its clinical applicability. Previous studies have focused primarily on the accurate estimation of the motion parameters employing a single step 3D interpolation at each individual time frame to recover a motion-free 4D fMRI image. Using only information from a 3D spatial neighborhood neglects the temporal structure of fMRI and useful information from neighboring timepoints. Her…
In an inhomogeneously illuminated photoacoustic image, important information like vascular geometry is not readily available when only the initial pressure is reconstructed. To obtain the desired information, algorithms for image segmentation are often applied as a post-processing step. In this work, we propose to jointly acquire the photoacoustic reconstruction and segmentation, by modifying a recently developed partially learned algorithm based on a convolutional neural network. We investigate the stability of the algorithm against changes in initial pressures and photoacoustic system settin…
Low-dose CT image reconstruction has been a popular research topic in recent years. A typical reconstruction method based on post-log measurements is called penalized weighted-least squares (PWLS). Due to the underlying limitations of the post-log statistical model, the PWLS reconstruction quality is often degraded in low-dose scans. This paper investigates a shifted-Poisson (SP) model based likelihood function that uses the pre-log raw measurements that better represents the measurement statistics, together with a data-driven regularizer exploiting a Union of Learned TRAnsforms (SPULTRA). Bot…
Video reconstruction from a single motion-blurred image is a challenging problem, which can enhance the capabilities of existing cameras. Recently, several works addressed this task using conventional imaging and deep learning. Yet, such purely-digital methods are inherently limited, due to direction ambiguity and noise sensitivity. Some works proposed to address these limitations using non-conventional image sensors, however, such sensors are extremely rare and expensive. To circumvent these limitations with simpler means, we propose a hybrid optical-digital method for video reconstruction th…
Biomedical Imaging Research Institute, Cedars-Sinai Medical Center（西达赛奈医疗中心生物医学影像研究所）；UCLA（加州大学洛杉矶分校）
Non-Cartesian sampling with subspace-constrained image reconstruction is a popular approach to dynamic MRI, but slow iterative reconstruction limits its clinical application. Data-consistent (DC) deep learning can accelerate reconstruction with good image quality, but has not been formulated for non-Cartesian subspace imaging. In this study, we propose a DC non-Cartesian deep subspace learning framework for fast, accurate dynamic MR image reconstruction. Four novel DC formulations are developed and evaluated: two gradient decent approaches, a directly solved approach, and a conjugate gradient …
Bilkent University（比尔肯特大学）；National Magnetic Resonance Research Center, Bilkent University（比尔肯特大学国家磁共振研究中心）
Supervised reconstruction models are characteristically trained on matched pairs of undersampled and fully-sampled data to capture an MRI prior, along with supervision regarding the imaging operator to enforce data consistency. To reduce supervision requirements, the recent deep image prior framework instead conjoins untrained MRI priors with the imaging operator during inference. Yet, canonical convolutional architectures are suboptimal in capturing long-range relationships, and priors based on randomly initialized networks may yield suboptimal performance. To address these limitations, here …
Computed tomography (CT) uses X-ray measurements taken from sensors around the body to generate tomographic images of the human body. Conventional reconstruction algorithms can be used if the X-ray data are adequately sampled and of high quality; however, concerns such as reducing dose to the patient, or geometric limitations on data acquisition, may result in low quality or incomplete data. Images reconstructed from these data using conventional methods are of poor quality, due to noise and other artifacts. The aim of this study is to train a single neural network to reconstruct high-quality …
Purpose: The goal of this study is to develop a novel deep learning (DL) based reconstruction framework to improve the digital breast tomosynthesis (DBT) imaging performance. Methods: In this work, the DIR-DBTnet is developed for DBT image reconstruction by unrolling the standard iterative reconstruction algorithm within the deep learning framework. In particular, such network learns the regularizer and the iteration parameters automatically through network training with a large amount of simulated DBT data. Afterwards, both numerical and experimental data are used to evaluate its performance.…
Middle East Technical University (METU)（中东技术大学）；ASELSAN Research Center（ASELSAN研究中心）
Classical phase retrieval problem is the recovery of a constrained image from the magnitude of its Fourier transform. Although there are several well-known phase retrieval algorithms including the hybrid input-output (HIO) method, the reconstruction performance is generally sensitive to initialization and measurement noise. Recently, deep neural networks (DNNs) have been shown to provide state-of-the-art performance in solving several inverse problems such as denoising, deconvolution, and superresolution. In this work, we develop a phase retrieval algorithm that utilizes two DNNs together with…
Iterative self-consistent parallel imaging reconstruction (SPIRiT) is an effective self-calibrated reconstruction model for parallel magnetic resonance imaging (PMRI). The joint L1 norm of wavelet coefficients and joint total variation (TV) regularization terms are incorporated into the SPIRiT model to improve the reconstruction performance. The simultaneous two-directional low-rankness (STDLR) in k-space data is incorporated into SPIRiT to realize improved reconstruction. Recent methods have exploited the nonlocal self-similarity (NSS) of images by imposing nonlocal low-rankness of similar pa…
Cross-Distribution Diffusion Priors-Driven Iterative Reconstruction for Sparse-View CT
Department of Electrical and Computer Engineering, University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校电子与计算机工程系）；School of Biomedical Engineering, Sun Yat-Sen University（中山大学生物医学工程学院）
Physikalisch-Technische Bundesanstalt (PTB)（德国联邦物理技术研究院）；University of Innsbruck（因斯布鲁克大学）；King’s College London（伦敦国王学院）；Technical University of Berlin（柏林工业大学）
Purpose: Iterative Convolutional Neural Networks (CNNs) which resemble unrolled learned iterative schemes have shown to consistently deliver state-of-the-art results for image reconstruction problems across different imaging modalities. However, because these methodes include the forward model in the architecture, their applicability is often restricted to either relatively small reconstruction problems or to problems with operators which are computationally cheap to compute. As a consequence, they have so far not been applied to dynamic non-Cartesian multi-coil reconstruction problems. Method…
Physikalisch-Technische Bundesanstalt（德国联邦物理技术研究院）；Charité - Universitätsmedizin Berlin（柏林夏里特医学院）；King’s College London（伦敦国王学院）；Technical University of Berlin（柏林工业大学）；University of Innsbruck（因斯布鲁克大学）
The concept of sparsity has been extensively applied for regularization in image reconstruction. Typically, sparsifying transforms are either pre-trained on ground-truth images or adaptively trained during the reconstruction. Thereby, learning algorithms are designed to minimize some target function which encodes the desired properties of the transform. However, this procedure ignores the subsequently employed reconstruction algorithm as well as the physical model which is responsible for the image formation process. Iterative neural networks - which contain the physical model - can overcome t…
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Reconstruction of few-view x-ray Computed Tomography (CT) data is a highly ill-posed problem. It is often used in applications that require low radiation dose in clinical CT, rapid industrial scanning, or fixed-gantry CT. Existing analytic or iterative algorithms generally produce poorly reconstructed images, severely deteriorated by artifacts and noise, especially when the number of x-ray projections is considerably low. This paper presents a deep network-driven approach to address extreme few-view CT by incorporating convolutional neural network-based inference into state-of-the-art iterativ…
Model-Based Image Reconstruction (MBIR) methods significantly enhance the quality of computed tomographic (CT) reconstructions relative to analytical techniques, but are limited by high computational cost. In this paper, we propose a multi-agent consensus equilibrium (MACE) algorithm for distributing both the computation and memory of MBIR reconstruction across a large number of parallel nodes. In MACE, each node stores only a sparse subset of views and a small portion of the system matrix, and each parallel node performs a local sparse-view reconstruction, which based on repeated feedback fro…
Spectral domain optical coherence tomography (OCT) offers high resolution multidimensional imaging, but generally suffers from defocussing, intensity falloff and shot noise, causing artifacts and image degradation along the imaging depth. In this work, we develop an iterative statistical reconstruction technique, based upon the interferometric synthetic aperture microscopy (ISAM) model with additive noise, to actively compensate for these effects. For the ISAM re-sampling, we use a non uniform FFT with Kaiser-Bessel interpolation, offering efficiency and high accuracy. We then employ an accele…
Department of Computer Science and Engineering, Washington University in St. Louis（圣路易斯华盛顿大学计算机科学与工程系）；Department of Electrical and Systems Engineering, Washington University in St. Louis（圣路易斯华盛顿大学电气与系统工程系）；Department of Bioengineering, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校生物工程系）
Medical image reconstruction is typically an ill-posed inverse problem. In order to address such ill-posed problems, the prior distribution of the sought after object property is usually incorporated by means of some sparsity-promoting regularization. Recently, prior distributions for images estimated using generative adversarial networks (GANs) have shown great promise in regularizing some of these image reconstruction problems. In this work, we apply an image-adaptive GAN-based reconstruction method (IAGAN) to reconstruct high fidelity images from incomplete medical imaging data. It is obser…
Shanghai Jiao Tong University（上海交通大学）；Michigan State University（密歇根州立大学）
Signal models based on sparse representation have received considerable attention in recent years. Compared to synthesis dictionary learning, sparsifying transform learning involves highly efficient sparse coding and operator update steps. In this work, we propose a Multi-layer Residual Sparsifying Transform (MRST) learning model wherein the transform domain residuals are jointly sparsified over layers. In particular, the transforms for the deeper layers exploit the more intricate properties of the residual maps. We investigate the application of the learned MRST model for low-dose CT reconstr…
Reconstruction of digital breast tomosynthesis is a challenging problem due to the limited angle data available in such systems. Due to memory limitations, deep learning-based methods can help improve these reconstructions, but can not (yet) attain sufficiently high resolution. In addition to this practical issue, questions remain on the possibility of such models introducing 'ghost' information from the training data that is not compatible with the projection data. To take advantage of some of the benefits of deep learning-based reconstructions while avoiding these limitations, we propose to …
Regularization by denoising (RED) is an image reconstruction framework that uses an image denoiser as a prior. Recent work has shown the state-of-the-art performance of RED with learned denoisers corresponding to pre-trained convolutional neural nets (CNNs). In this work, we propose to broaden the current denoiser-centric view of RED by considering priors corresponding to networks trained for more general artifact-removal. The key benefit of the proposed family of algorithms, called regularization by artifact-removal (RARE), is that it can leverage priors learned on datasets containing only un…
A new computational imaging method to reconstruct the complex wave-field is reported. Due to the existence of zero frequency component, the measured signal by amplitude modulation of pupil has a spectrum similar to the one of off-axis hologram. The mathematical analogy between them is established in this paper. Based on this observation and analyticity of band-limited signal under any diffraction-limited system, an algorithm from Kramers-Kronig (KK) relations is utilized to recover the phase information only from the intensity patterns. From the sensing side, only two measurements are required…
