Magnetic Resonance Imaging (MRI) is a non-invasive diagnostic tool that provides excellent soft-tissue contrast without the use of ionizing radiation. Compared to other clinical imaging modalities (e.g., CT or ultrasound), however, the data acquisition process for MRI is inherently slow, which motivates undersampling and thus drives the need for accurate, efficient reconstruction methods from undersampled datasets. In this article, we describe the use of "plug-and-play" (PnP) algorithms for MRI image recovery. We first describe the linearly approximated inverse problem encountered in MRI. Then we review several PnP methods, where the unifying commonality is to iteratively call a denoising subroutine as one step of a larger optimization-inspired algorithm. Next, we describe how the result of the PnP method can be interpreted as a solution to an equilibrium equation, allowing convergence analysis from the equilibrium perspective. Finally, we present illustrative examples of PnP methods applied to MRI image recovery.
Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico（高等技术学院）
The recently proposed plug-and-play (PnP) framework allows leveraging recent developments in image denoising to tackle other, more involved, imaging inverse problems. In a PnP method, a black-box denoiser is plugged into an iterative algorithm, taking the place of a formal denoising step that corresponds to the proximity operator of some convex regularizer. While this approach offers flexibility and excellent performance, convergence of the resulting algorithm may be hard to analyze, as most state-of-the-art denoisers lack an explicit underlying objective function. In this paper, we propose a …
In this work we propose an efficient stochastic plug-and-play (PnP) algorithm for imaging inverse problems. The PnP stochastic gradient descent methods have been recently proposed and shown improved performance in some imaging applications over standard deterministic PnP methods. However, current stochastic PnP methods need to frequently compute the image denoisers which can be computationally expensive. To overcome this limitation, we propose a new stochastic PnP-ADMM method which is based on introducing stochastic gradient descent inner-loops within an inexact ADMM framework. We provide the …
Beijing Institute of Technology（北京理工大学）；University of Cambridge（剑桥大学）
Plug-and-play (PnP) is a non-convex framework that combines ADMM or other proximal algorithms with advanced denoiser priors. Recently, PnP has achieved great empirical success, especially with the integration of deep learning-based denoisers. However, a key problem of PnP based approaches is that they require manual parameter tweaking. It is necessary to obtain high-quality results across the high discrepancy in terms of imaging conditions and varying scene content. In this work, we present a tuning-free PnP proximal algorithm, which can automatically determine the internal parameters includin…
MPISuperRes-PnP: A Super-Resolution Zero-Shot Plug-and-Play Reconstruction Algorithm for Magnetic Particle Imaging
Institute for the Protection of Terrestrial Infrastructures, German Aerospace Center (DLR)（德国航空航天中心陆地基础设施保护研究所）；Hochschule Darmstadt（达姆施塔特应用科学大学）；Data Science Institute, European University of Technology（欧洲技术大学数据科学研究所）；Technische Hochschule Würzburg-Schweinfurt（维尔茨堡-施韦因富特应用技术大学）
Snapshot compressive imaging (SCI) aims to capture the high-dimensional (usually 3D) images using a 2D sensor (detector) in a single snapshot. Though enjoying the advantages of low-bandwidth, low-power and low-cost, applying SCI to large-scale problems (HD or UHD videos) in our daily life is still challenging. The bottleneck lies in the reconstruction algorithms; they are either too slow (iterative optimization algorithms) or not flexible to the encoding process (deep learning based end-to-end networks). In this paper, we develop fast and flexible algorithms for SCI based on the plug-and-play …
Beijing Institute of Technology（北京理工大学）；University of Cambridge（剑桥大学）；Shanghai Jiao Tong University（上海交通大学）；Beijing Normal University（北京师范大学）
Plug-and-Play (PnP) is a non-convex optimization framework that combines proximal algorithms, for example, the alternating direction method of multipliers (ADMM), with advanced denoising priors. Over the past few years, great empirical success has been obtained by PnP algorithms, especially for the ones that integrate deep learning-based denoisers. However, a key challenge of PnP approaches is the need for manual parameter tweaking as it is essential to obtain high-quality results across the high discrepancy in imaging conditions and varying scene content. In this work, we present a class of t…
Alternating direction method of multiplier (ADMM) is a widely used algorithm for solving constrained optimization problems in image restoration. Among many useful features, one critical feature of the ADMM algorithm is its modular structure which allows one to plug in any off-the-shelf image denoising algorithm for a subproblem in the ADMM algorithm. Because of the plug-in nature, this type of ADMM algorithms is coined the name "Plug-and-Play ADMM". Plug-and-Play ADMM has demonstrated promising empirical results in a number of recent papers. However, it is unclear under what conditions and by …
Washington University in St. Louis（圣路易斯华盛顿大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
Plug-and-play priors (PnP) is a powerful framework for regularizing imaging inverse problems by using advanced denoisers within an iterative algorithm. Recent experimental evidence suggests that PnP algorithms achieve state-of-the-art performance in a range of imaging applications. In this paper, we introduce a new online PnP algorithm based on the iterative shrinkage/thresholding algorithm (ISTA). The proposed algorithm uses only a subset of measurements at every iteration, which makes it scalable to very large datasets. We present a new theoretical convergence analysis, for both batch and on…
A Plug-and-Play Framework for Volumetric Light-Sheet Image Reconstruction
Department of Mathematics（数学系）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Department of Bioengineering（生物工程系）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
In plug-and-play image restoration, the regularization is performed using powerful denoisers such as nonlocal means (NLM) or BM3D. This is done within the framework of alternating direction method of multipliers (ADMM), where the regularization step is formally replaced by an off-the-shelf denoiser. Each plug-and-play iteration involves the inversion of the forward model followed by a denoising step. In this paper, we present a couple of ideas for improving the efficiency of the inversion and denoising steps. First, we propose to use linearized ADMM, which generally allows us to perform the in…
PLUS: Plug-and-Play Enhanced Liver Lesion Diagnosis Model on Non-Contrast CT Scans
School of Biomedical Engineering, Tsinghua University（清华大学生物医学工程学院）；DAMO Academy, Alibaba Group（阿里集团达摩院）；Hupan Lab（汇看实验室）；Department of Radiology, Shengjing Hospital of China Medical University（中国医科大学附属盛京医院放射科）；Department of Hepatobiliary and Pancreatic Surgery, First Affiliated Hospital of Zhejiang University（浙江大学医学院附属第一医院肝胆外科）
Bell Labs（贝尔实验室）；Massachusetts Institute of Technology（麻省理工学院）；Tsinghua University（清华大学）
We consider the reconstruction problem of video snapshot compressive imaging (SCI), which captures high-speed videos using a low-speed 2D sensor (detector). The underlying principle of SCI is to modulate sequential high-speed frames with different masks and then these encoded frames are integrated into a snapshot on the sensor and thus the sensor can be of low-speed. On one hand, video SCI enjoys the advantages of low-bandwidth, low-power and low-cost. On the other hand, applying SCI to large-scale problems (HD or UHD videos) in our daily life is still challenging and one of the bottlenecks li…
Continuous protocols for cardiac magnetic resonance imaging enable sampling of the cardiac anatomy simultaneously resolved into cardiac phases. To avoid respiration artifacts, associated motion during the scan has to be compensated for during reconstruction. In this paper, we propose a sampling adaption to acquire 2-D respiration information during a continuous scan. Further, we develop a pipeline to extract the different respiration states from the acquired signals, which are used to reconstruct data from one respiration phase. Our results show the benefit of the proposed workflow on the imag…
Biomediq A/S（Biomediq公司）；University of Copenhagen（哥本哈根大学）
Scanner drift is a well-known magnetic resonance imaging (MRI) artifact characterized by gradual signal degradation and scan intensity changes over time. In addition, hardware and software updates may imply abrupt changes in signal. The combined effects are particularly challenging for automatic image analysis methods used in longitudinal studies. The implication is increased measurement variation and a risk of bias in the estimations (e.g. in the volume change for a structure). We proposed two quite different approaches for scanner drift normalization and demonstrated the performance for segm…
East China Normal University（华东师范大学）；The University of Hong Kong（香港大学）
Aiming at high-dimensional (HD) data acquisition and analysis, snapshot compressive imaging (SCI) obtains the 2D compressed measurement of HD data with optical imaging systems and reconstructs HD data using compressive sensing algorithms. While the Plug-and-Play (PnP) framework offers an emerging solution to SCI reconstruction, its intrinsic denoising process is still a challenging problem. Unfortunately, existing denoisers in the PnP framework either suffer limited performance or require extensive training data. In this paper, we propose an efficient and effective shallow-learning-based algor…
University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）；Hangzhou Dianzi University（杭州电子科技大学）；Shandong University（山东大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Leicester（莱斯特大学）；Communication University of Zhejiang（浙江传媒学院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Skull stripping is a crucial prerequisite step in the analysis of brain magnetic resonance images (MRI). Although many excellent works or tools have been proposed, they suffer from low generalization capability. For instance, the model trained on a dataset with specific imaging parameters cannot be well applied to other datasets with different imaging parameters. Especially, for the lifespan datasets, the model trained on an adult dataset is not applicable to an infant dataset due to the large domain difference. To address this issue, numerous methods have been proposed, where domain adaptatio…
Purdue University（普渡大学）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Air Force Research Laboratory（美国空军研究实验室）
Many material and biological samples in scientific imaging are characterized by non-local repeating structures. These are studied using scanning electron microscopy and electron tomography. Sparse sampling of individual pixels in a 2D image acquisition geometry, or sparse sampling of projection images with large tilt increments in a tomography experiment, can enable high speed data acquisition and minimize sample damage caused by the electron beam. In this paper, we present an algorithm for electron tomographic reconstruction and sparse image interpolation that exploits the non-local redundanc…
Cardiac magnetic resonance imaging (CMR) is a noninvasive imaging modality that provides a comprehensive evaluation of the cardiovascular system. The clinical utility of CMR is hampered by long acquisition times, however. In this work, we propose and validate a plug-and-play (PnP) method for CMR reconstruction from undersampled multi-coil data. To fully exploit the rich image structure inherent in CMR, we pair the PnP framework with a deep learning (DL)-based denoiser that is trained using spatiotemporal patches from high-quality, breath-held cardiac cine images. The resulting "PnP-DL" method …
School of Biomedical Engineering（生物医学工程学院）；Universidad Autónoma de Ciudad Juárez（华雷斯自治大学）
The brain is a complex organ controlling cognitive process and physical functions. Tumors in the brain are accelerated cell growths affecting the normal function and processes in the brain. MRI scans provides detailed images of the body being one of the most common tests to diagnose brain tumors. The process of segmentation of brain tumors from magnetic resonance imaging can provide a valuable guide for diagnosis, treatment planning and prediction of results. Here we consider the problem brain tumor segmentation using a Deep learning architecture for use in tumor segmentation. Although the pro…
Despite the recent advances of deep learning algorithms in medical imaging, the automatic segmentation algorithms for kidneys in MRI exams are still scarce. Automated segmentation of kidneys in Magnetic Resonance Imaging (MRI) exams are important for enabling radiomics and machine learning analysis of renal disease. In this work, we propose to use the popular Mask R-CNN for the automatic segmentation of kidneys in coronal T2-weighted Fast Spin Eco slices of 100 MRI exams. We propose the morphological operations as post-processing to further improve the performance of Mask R-CNN for this task. …
This dissertation is devoted to provide advanced nonconvex nonsmooth variational models of (Magnetic Resonance Image) MRI reconstruction, efficient learnable image reconstruction algorithms and parameter training algorithms that improve the accuracy and robustness of the optimization-based deep learning methods for compressed sensing MRI reconstruction and synthesis. The first part introduces a novel optimization based deep neural network whose architecture is inspired by proximal gradient descent for solving a variational model. The second part is a substantial extension of the preliminary wo…
Harvard Medical School（哈佛医学院）；Boston Children’s Hospital（波士顿儿童医院）
Fast data acquisition in Magnetic Resonance Imaging (MRI) is vastly in demand and scan time directly depends on the number of acquired k-space samples. Conventional MRI reconstruction methods for fast MRI acquisition mostly relied on different regularizers which represent analytical models of sparsity. However, recent data-driven methods based on deep learning has resulted in promising improvements in image reconstruction algorithms. In this paper, we propose a deep plug-and-play prior framework for parallel MRI reconstruction problems which utilize a deep neural network (DNN) as an advanced d…
Bayesian imaging using Plug & Play priors: when Langevin meets Tweedie
University of Oxford（牛津大学）；Institut Universitaire de France（法国大学研究院）；Heriot-Watt University（赫瑞瓦特大学）；Maxwell Institute for Mathematical Sciences（麦克斯韦数学科学研究所）
本文针对贝叶斯成像中 Plug & Play (PnP) 先验方法缺乏收敛证明的问题，提出了两种可证明收敛的算法 PnP-ULA 和 PnP-SGD，并建立了详细的收敛保证，适用于图像去模糊、修复和去噪等任务。
Plug-and-Play methods constitute a class of iterative algorithms for imaging problems where regularization is performed by an off-the-shelf denoiser. Although Plug-and-Play methods can lead to tremendous visual performance for various image problems, the few existing convergence guarantees are based on unrealistic (or suboptimal) hypotheses on the denoiser, or limited to strongly convex data terms. In this work, we propose a new type of Plug-and-Play methods, based on half-quadratic splitting, for which the denoiser is realized as a gradient descent step on a functional parameterized by a deep…
Intel Labs（英特尔实验室）；University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）
Magnetic resonance imaging is capable of producing volumetric images without ionizing radiation. Nonetheless, long acquisitions lead to prohibitively long exams. Compressed sensing (CS) can enable faster scanning via sub-sampling with reduced artifacts. However, CS requires significantly higher reconstruction computation, limiting current clinical applications to 2D/3D or limited-resolution dynamic imaging. Here we analyze the practical limitations to T2 Shuffling, a four-dimensional CS-based acquisition, which provides sharp 3D-isotropic-resolution and multi-contrast images in a single scan. …
We introduce a novel energy formulation for Plug- and-Play (PnP) image recovery. Traditional PnP methods that use a convolutional neural network (CNN) do not have an energy based formulation. The primary focus of this work is to introduce an energy-based PnP formulation, which relies on a CNN that learns the log of the image prior from training data. The score function is evaluated as the gradient of the energy model, which resembles a UNET with shared encoder and decoder weights. The proposed score function is thus constrained to a conservative vector field, which is the key difference with c…
University of Southampton（南安普顿大学）；University Hospital Southampton NHS Foundation Trust（南安普顿大学医院NHS基金会信托）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
A desire to achieve large medical imaging datasets keeps increasing as machine learning algorithms, parallel computing, and hardware technology evolve. Accordingly, there is a growing demand in pooling data from multiple clinical and academic institutes to enable large-scale clinical or translational research studies. Magnetic resonance imaging (MRI) is a frequently used, non-invasive imaging modality. However, constructing a big MRI data repository has multiple challenges related to privacy, data size, DICOM format, logistics, and non-standardized images. Not only building the data repository…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Massachusetts Institute of Technology（麻省理工学院）
Image corruption by motion artifacts is an ingrained problem in Magnetic Resonance Imaging (MRI). In this work, we propose a neural network-based regularization term to enhance Autofocusing, a classic optimization-based method to remove motion artifacts. The method takes the best of both worlds: the optimization-based routine iteratively executes the blind demotion and deep learning-based prior penalizes for unrealistic restorations and speeds up the convergence. We validate the method on three models of motion trajectories, using synthetic and real noisy data. The method proves resilient to n…
Washington University in St. Louis（圣路易斯华盛顿大学）；University of California, Riverside（加州大学河滨分校）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
The plug-and-play priors (PnP) and regularization by denoising (RED) methods have become widely used for solving inverse problems by leveraging pre-trained deep denoisers as image priors. While the empirical imaging performance and the theoretical convergence properties of these algorithms have been widely investigated, their recovery properties have not previously been theoretically analyzed. We address this gap by showing how to establish theoretical recovery guarantees for PnP/RED by assuming that the solution of these methods lies near the fixed-points of a deep neural network. We also pre…
ShanghaiTech University（上海科技大学）；University of Zürich（苏黎世大学）；ETH Zürich（苏黎世联邦理工学院）
Multi-spectral optoacoustic tomography (MSOT) is an emerging optical imaging method providing multiplex molecular and functional information from the rodent brain. It can be greatly augmented by magnetic resonance imaging (MRI) that offers excellent soft-tissue contrast and high-resolution brain anatomy. Nevertheless, registration of multi-modal images remains challenging, chiefly due to the entirely different image contrast rendered by these modalities. Previously reported registration algorithms mostly relied on manual user-dependent brain segmentation, which compromised data interpretation …
Technical University of Munich（慕尼黑工业大学）；Imperial College London（帝国理工学院）；University Hospital of Tuebingen（图宾根大学医院）；University of Minnesota（明尼苏达大学）；New York University（纽约大学）；Friedrich-Alexander University Erlangen（埃尔朗根-纽伦堡大学）
Physics-driven deep learning methods have emerged as a powerful tool for computational magnetic resonance imaging (MRI) problems, pushing reconstruction performance to new limits. This article provides an overview of the recent developments in incorporating physics information into learning-based MRI reconstruction. We consider inverse problems with both linear and non-linear forward models for computational MRI, and review the classical approaches for solving these. We then focus on physics-driven deep learning approaches, covering physics-driven loss functions, plug-and-play methods, generat…
