JoJoNet: Joint-contrast and Joint-sampling-and-reconstruction Network for Multi-contrast MRI
University of Georgia（佐治亚大学）；United Imaging Intelligence（联影智能）；ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能有限公司）
Lin Zhao, Xiao Chen, Eric Z. Chen, Yikang Liu, Dinggang Shen, Terrence Chen, Shanhui Sun
Multi-contrast Magnetic Resonance Imaging (MRI) generates multiple medical images with rich and complementary information for routine clinical use; however, it suffers from a long acquisition time. Recent works for accelerating MRI, mainly designed for single contrast, may not be optimal for multi-contrast scenario since the inherent correlations among the multi-contrast images are not exploited. In addition, independent reconstruction of each contrast usually does not translate to optimal performance of downstream tasks. Motivated by these aspects, in this paper we design an end-to-end framework for accelerating multi-contrast MRI which simultaneously optimizes the entire MR imaging workflow including sampling, reconstruction and downstream tasks to achieve the best overall outcomes. The proposed framework consists of a sampling mask generator for each image contrast and a reconstructor exploiting the inter-contrast correlations with a recurrent structure which enables the information sharing in a holistic way. The sampling mask generator and the reconstructor are trained jointly across the multiple image contrasts. The acceleration ratio of each image contrast is also learnable and can be driven by a downstream task performance. We validate our approach on a multi-contrast brain dataset and a multi-contrast knee dataset. Experiments show that (1) our framework consistently outperforms the baselines designed for single contrast on both datasets; (2) our newly designed recurrent reconstruction network effectively improves the reconstruction quality for multi-contrast images; (3) the learnable acceleration ratio improves the downstream task performance significantly. Overall, this work has potentials to open up new avenues for optimizing the entire multi-contrast MR imaging workflow.
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Tsinghua University（清华大学）
Compressed Sensing MRI (CS-MRI) aims at reconstructing de-aliased images from sub-Nyquist sampling k-space data to accelerate MR Imaging, thus presenting two basic issues, i.e., where to sample and how to reconstruct. To deal with both problems simultaneously, we propose a novel end-to-end Probabilistic Under-sampling and Explicable Reconstruction neTwork, dubbed PUERT, to jointly optimize the sampling pattern and the reconstruction network. Instead of learning a deterministic mask, the proposed sampling subnet explores an optimal probabilistic sub-sampling pattern, which describes independent…
University of Cambridge（剑桥大学）；ShanghaiTech University（上海科技大学）；Zhejiang University of Technology（浙江工业大学）；Northwest University（西北大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）
Recent studies on T1-assisted MRI reconstruction for under-sampled images of other modalities have demonstrated the potential of further accelerating MRI acquisition of other modalities. Most of the state-of-the-art approaches have achieved improvement through the development of network architectures for fixed under-sampling patterns, without fully exploiting the complementary information between modalities. Although existing under-sampling pattern learning algorithms can be simply modified to allow the fully-sampled T1-weighted MR image to assist the pattern learning, no significant improveme…
The University of Queensland（昆士兰大学）；Griffith University（格里菲斯大学）
MRI images of the same subject in different contrasts contain shared information, such as the anatomical structure. Utilizing the redundant information amongst the contrasts to sub-sample and faithfully reconstruct multi-contrast images could greatly accelerate the imaging speed, improve image quality and shorten scanning protocols. We propose an algorithm that generates the optimised sampling pattern and reconstruction scheme of one contrast (e.g. T2-weighted image) when images with different contrast (e.g. T1-weighted image) have been acquired. The proposed algorithm achieves increased PSNR …
Compressed sensing applied to magnetic resonance imaging (MRI) allows to reduce the scanning time by enabling images to be reconstructed from highly undersampled data. In this paper, we tackle the problem of designing a sampling mask for an arbitrary reconstruction method and a limited acquisition budget. Namely, we look for an optimal probability distribution from which a mask with a fixed cardinality is drawn. We demonstrate that this problem admits a compactly supported solution, which leads to a deterministic optimal sampling mask. We then propose a stochastic greedy algorithm that (i) pro…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；California Institute of Technology（加州理工学院）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）
Accelerated MRI shortens acquisition time by subsampling in the measurement $κ$-space. Recovering a high-fidelity anatomical image from subsampled measurements requires close cooperation between two components: (1) a sampler that chooses the subsampling pattern and (2) a reconstructor that recovers images from incomplete measurements. In this paper, we leverage the sequential nature of MRI measurements, and propose a fully differentiable framework that jointly learns a sequential sampling policy simultaneously with a reconstruction strategy. This co-designed framework is able to adapt during a…
Facebook AI Research（Facebook人工智能研究）；fastMRI team（fastMRI团队）
Deep learning approaches to accelerated MRI take a matrix of sampled Fourier-space lines as input and produce a spatial image as output. In this work we show that by careful choice of the offset used in the sampling procedure, the symmetries in k-space can be better exploited, producing higher quality reconstructions than given by standard equally-spaced samples or randomized samples motivated by compressed sensing.
Cornell University（康奈尔大学）；Weill Medical College of Cornell University（康奈尔大学威尔医学院）
The previously established LOUPE (Learning-based Optimization of the Under-sampling Pattern) framework for optimizing the k-space sampling pattern in MRI was extended in three folds: firstly, fully sampled multi-coil k-space data from the scanner, rather than simulated k-space data from magnitude MR images in LOUPE, was retrospectively under-sampled to optimize the under-sampling pattern of in-vivo k-space data; secondly, binary stochastic k-space sampling, rather than approximate stochastic k-space sampling of LOUPE during training, was applied together with a straight-through (ST) estimator …
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）
We developed a new joint probabilistic segmentation and image distribution matching generative adversarial network (PSIGAN) for unsupervised domain adaptation (UDA) and multi-organ segmentation from magnetic resonance (MRI) images. Our UDA approach models the co-dependency between images and their segmentation as a joint probability distribution using a new structure discriminator. The structure discriminator computes structure of interest focused adversarial loss by combining the generated pseudo MRI with probabilistic segmentations produced by a simultaneously trained segmentation sub-networ…
University of Iowa（爱荷华大学）；Philips HealthCare（飞利浦医疗）；University of Wisconsin（威斯康星大学）；GE HealthCare（通用电气医疗）
Free-breathing cardiac MRI schemes are emerging as competitive alternatives to breath-held cine MRI protocols, enabling applicability to pediatric and other population groups that cannot hold their breath. Because the data from the slices are acquired sequentially, the cardiac/respiratory motion patterns may be different for each slice; current free-breathing approaches perform independent recovery of each slice. In addition to not being able to exploit the inter-slice redundancies, manual intervention or sophisticated post-processing methods are needed to align the images post-recovery for qu…
The MRI Scanner as a Diagnostic: Image-less Active Sampling
School of Engineering, The University of Edinburgh（爱丁堡大学工程学院）；Krannert Cardiovascular Research Center, Indiana University School of Medicine（印第安纳大学医学院克兰纳特心血管研究中心）；The Alan Turing Institute（阿兰·图灵研究所）
University of Cambridge（剑桥大学）；Rice University（莱斯大学）
The Magnetic Resonance Imaging (MRI) processing chain starts with a critical acquisition stage that provides raw data for reconstruction of images for medical diagnosis. This flow usually includes a near-lossless data compression stage that enables digital storage and/or transmission in binary formats. In this work we propose a framework for joint optimization of the MRI reconstruction and lossy compression, producing compressed representations of medical images that achieve improved trade-offs between quality and bit-rate. Moreover, we demonstrate that lossy compression can even improve the r…
Hybrid Learning: A Novel Combination of Self-Supervised and Supervised Learning for Joint MRI Reconstruction and Denoising in Low-Field MRI
New York University Grossman School of Medicine（纽约大学格罗斯曼医学院）；NYU Tandon School of Engineering（纽约大学坦登工程学院）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）
The Australian National University（澳大利亚国立大学）；University College London（伦敦大学学院）
This paper proposes a multi-shell sampling scheme and corresponding transforms for the accurate reconstruction of the diffusion signal in diffusion MRI by expansion in the spherical polar Fourier (SPF) basis. The sampling scheme uses an optimal number of samples, equal to the degrees of freedom of the band-limited diffusion signal in the SPF domain, and allows for computationally efficient reconstruction. We use synthetic data sets to demonstrate that the proposed scheme allows for greater reconstruction accuracy of the diffusion signal than the multi-shell sampling schemes obtained using the …
Deep learning (DL) has shown promise for faster, high quality accelerated MRI reconstruction. However, supervised DL methods depend on extensive amounts of fully-sampled (labeled) data and are sensitive to out-of-distribution (OOD) shifts, particularly low signal-to-noise ratio (SNR) acquisitions. To alleviate this challenge, we propose Noise2Recon, a model-agnostic, consistency training method for joint MRI reconstruction and denoising that can use both fully-sampled (labeled) and undersampled (unlabeled) scans in semi-supervised and self-supervised settings. With limited or no labeled traini…
University of California, Santa Barbara（加州大学圣塔芭芭拉分校）；Oden Institute for Computational Engineering and Sciences（奥登计算工程与科学研究所）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
Magnetic resonance imaging (MRI) is a powerful imaging modality that revolutionizes medicine and biology. The imaging speed of high-dimensional MRI is often limited, which constrains its practical utility. Recently, low-rank tensor models have been exploited to enable fast MR imaging with sparse sampling. Most existing methods use some pre-defined sampling design, and active sensing has not been explored for low-rank tensor imaging. In this paper, we introduce an active low-rank tensor model for fast MR imaging. We propose an active sampling method based on a Query-by-Committee model, making u…
Purpose: To develop a synergistic image reconstruction framework that exploits multicontrast (MC), multicoil, and compressed sensing (CS) redundancies in magnetic resonance imaging (MRI). Approach: CS, MC acquisition, and parallel imaging (PI) have been individually well developed, but the combination of the three has not been equally well studied, much less the potential benefits of isotropy within such a setting. Inspired by total variation theory, we introduce an isotropic MC image regularizer and attain its full potential by integrating it into compressed MC multicoil MRI. A convex optimiz…
The need for fast acquisition and automatic analysis of MRI data is growing in the age of big data. Although compressed sensing magnetic resonance imaging (CS-MRI) has been studied to accelerate MRI by reducing k-space measurements, in current CS-MRI techniques MRI applications such as segmentation are overlooked when doing image reconstruction. In this paper, we test the utility of CS-MRI methods in automatic segmentation models and propose a unified deep neural network architecture called SegNetMRI which we apply to the combined CS-MRI reconstruction and segmentation problem. SegNetMRI is bu…
The volume estimation of brain regions from MRI data is a key problem in many clinical applications, where the acquisition of data at high spatial resolution is desirable. While parallel MRI and constrained image reconstruction algorithms can accelerate the scans, image reconstruction artifacts are inevitable, especially at high acceleration factors. We introduce a novel image domain deep-learning framework for calibrationless parallel MRI reconstruction, coupled with a segmentation network to improve image quality and to reduce the vulnerability of current segmentation algorithms to image art…
Joint PET-MRI Reconstruction with Diffusion Stochastic Differential Model
Inner Mongolia University（内蒙古大学）；Inner Mongolia Medical University（内蒙古医科大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Inner Mongolia Medical University Affiliated Hospital（内蒙古医科大学附属医院）
针对 PET-MRI 系统中 MRI 采集耗时长且 PET 信噪比低的问题，本文提出一种基于扩散随机微分方程的联合重建模型，通过学习两者的联合概率分布加速 MRI 并提升 PET 质量，在定性和定量上超越现有最先进方法。
Tel Aviv Medical Center（特拉维夫医疗中心）；Tel Aviv University（特拉维夫大学）
Purpose: Repeated brain MRI scans are performed in many clinical scenarios, such as follow up of patients with tumors and therapy response assessment. In this paper, the authors show an approach to utilize former scans of the patient for the acceleration of repeated MRI scans. Methods: The proposed approach utilizes the possible similarity of the repeated scans in longitudinal MRI studies. Since similarity is not guaranteed, sampling and reconstruction are adjusted during acquisition to match the actual similarity between the scans. The baseline MR scan is utilized both in the sampling stage, …
University of Florida（佛罗里达大学）；Georgia State University（佐治亚州立大学）
Generating multi-contrasts/modal MRI of the same anatomy enriches diagnostic information but is limited in practice due to excessive data acquisition time. In this paper, we propose a novel deep-learning model for joint reconstruction and synthesis of multi-modal MRI using incomplete k-space data of several source modalities as inputs. The output of our model includes reconstructed images of the source modalities and high-quality image synthesized in the target modality. Our proposed model is formulated as a variational problem that leverages several learnable modality-specific feature extract…
We propose to simultaneously learn to sample and reconstruct magnetic resonance images (MRI) to maximize the reconstruction quality given a limited sample budget, in a self-supervised setup. Unlike existing deep methods that focus only on reconstructing given data, thus being passive, we go beyond the current state of the art by considering both the data acquisition and the reconstruction process within a single deep-learning framework. As our network learns to acquire data, the network is active in nature. In order to do so, we simultaneously train two neural networks, one dedicated to recons…
NexOP: Joint Optimization of NEX-Aware k-space Sampling and Image Reconstruction for Low-Field MRI
Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology（电气与计算机工程系，技术学院–以色列理工学院）；May-Blum-Dahl Technion Human MRI Research Center, Technion - Israel Institute of Technology（May-Blum-Dahl技术学院人类MRI研究中心，技术学院–以色列理工学院）；Department of Biomedical Engineering, Technion – Israel Institute of Technology（生物医学工程系，技术学院–以色列理工学院）
Measured-Subspace Consistency: A Plug-and-Play Operator for Diffusion Posterior Sampling in Accelerated MRI Reconstruction
Interdisciplinary Program in Cancer Biology, Seoul National University College of Medicine（首尔国立大学医学院癌症生物学跨学科项目）；Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital（首尔国立大学医学院放射科，首尔国立大学医院）；Healthcare AI Research Institute, Seoul National University Hospital（首尔国立大学医院医疗人工智能研究机构）
University of Virginia（弗吉尼亚大学）；University of Virginia Health System（弗吉尼亚大学卫生系统）
Automated identification of myocardial scar from late gadolinium enhancement cardiac magnetic resonance images (LGE-CMR) is limited by image noise and artifacts such as those related to motion and partial volume effect. This paper presents a novel joint deep learning (JDL) framework that improves such tasks by utilizing simultaneously learned myocardium segmentations to eliminate negative effects from non-region-of-interest areas. In contrast to previous approaches treating scar detection and myocardium segmentation as separate or parallel tasks, our proposed method introduces a message passin…
Self-Supervised Joint Reconstruction and Denoising of T2-Weighted PROPELLER MRI of the Lungs at 0.55T
New York University Grossman School of Medicine（纽约大学格罗斯曼医学院）；New York University Tandon School of Engineering（纽约大学坦登工程学院）；Indiana University School of Medicine（印第安纳大学医学院）
Artificial Intelligence (Deep Learning(DL)/ Machine Learning(ML)) techniques are widely being used to address and overcome all kinds of ill-posed problems in medical imaging which was or in fact is seemingly impossible. Reducing gradient directions but harnessing high angular resolution(HAR) diffusion data in MR that retains clinical features is an important and challenging problem in the field. While the DL/ML approaches are promising, it is important to incorporate relevant context for the data to ensure that maximum prior information is provided for the AI model to infer the posterior. In t…
Monash Biomedical Imaging, Monash University（莫纳什大学生物医学成像中心）；Department of Electrical and Computer Systems Engineering, Monash University（莫纳什大学电气与计算机系统工程系）；School of Psychological Sciences, Monash University（莫纳什大学心理科学学院）；Department of Data Science and AI, Monash University（莫纳什大学数据科学与人工智能系）
Global correlations are widely seen in human anatomical structures due to similarity across tissues and bones. These correlations are reflected in magnetic resonance imaging (MRI) scans as a result of close-range proton density and T1/T2 parameters. Furthermore, to achieve accelerated MRI, k-space data are undersampled which causes global aliasing artifacts. Convolutional neural network (CNN) models are widely utilized for accelerated MRI reconstruction, but those models are limited in capturing global correlations due to the intrinsic locality of the convolution operation. The self-attention-…
Technical University Munich（慕尼黑工业大学）；Ludwig Maximilians University Munich（慕尼黑大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Alzheimer's disease (AD) is the most common cause of dementia. An early detection is crucial for slowing down the disease and mitigating risks related to the progression. While the combination of MRI and FDG-PET is the best image-based tool for diagnosis, FDG-PET is not always available. The reliable detection of Alzheimer's disease with only MRI could be beneficial, especially in regions where FDG-PET might not be affordable for all patients. To this end, we propose a multi-task method based on U-Net that takes T1-weighted MR images as an input to generate synthetic FDG-PET images and classif…
Britton Chance Center for Biomedical Photonics（布里顿·钱斯生物医学光子学研究中心）；Wuhan National Laboratory for Optoelectronics（武汉光电国家研究中心）；MoE Key Laboratory for Biomedical Photonics（教育部生物医学光子学重点实验室）；Collaborative Innovation Center for Biomedical Engineering（生物医学工程协同创新中心）；School of Engineering Sciences, Huazhong University of Science and Technology（华中科技大学工程科学学院）
Registration of brain MRI images requires to solve a deformation field, which is extremely difficult in aligning intricate brain tissues, e.g., subcortical nuclei, etc. Existing efforts resort to decomposing the target deformation field into intermediate sub-fields with either tiny motions, i.e., progressive registration stage by stage, or lower resolutions, i.e., coarse-to-fine estimation of the full-size deformation field. In this paper, we argue that those efforts are not mutually exclusive, and propose a unified framework for robust brain MRI registration in both progressive and coarse-to-…
University of Cambridge（剑桥大学）；Escuela Politécnica Nacional（厄瓜多尔国立理工学院）；University of Bath（巴斯大学）
The discovery of the theory of compressed sensing brought the realisation that many inverse problems can be solved even when measurements are "incomplete". This is particularly interesting in magnetic resonance imaging (MRI), where long acquisition times can limit its use. In this work, we consider the problem of learning a sparse sampling pattern that can be used to optimally balance acquisition time versus quality of the reconstructed image. We use a supervised learning approach, making the assumption that our training data is representative enough of new data acquisitions. We demonstrate th…
Fiber tractography is an important tool of computational neuroscience that enables reconstructing the spatial connectivity and organization of white matter of the brain. Fiber tractography takes advantage of diffusion Magnetic Resonance Imaging (dMRI) which allows measuring the apparent diffusivity of cerebral water along different spatial directions. Unfortunately, collecting such data comes at the price of reduced spatial resolution and substantially elevated acquisition times, which limits the clinical applicability of dMRI. This problem has been thus far addressed using two principal strat…
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）
Method: ProRSeg was trained using 5-fold cross-validation with 110 T2-weighted MRI acquired at 5 treatment fractions from 10 different patients, taking care that same patient scans were not placed in training and testing folds. Segmentation accuracy was measured using Dice similarity coefficient (DSC) and Hausdorff distance at 95th percentile (HD95). Registration consistency was measured using coefficient of variation (CV) in displacement of OARs. Ablation tests and accuracy comparisons against multiple methods were done. Finally, applicability of ProRSeg to segment cone-beam CT (CBCT) scans w…
Frequency-Hierarchical Active k-Space Sampling for Diagnostic MRI
Istanbul Technical University（伊斯坦布尔技术大学）；University of Toronto（多伦多大学）；Sunnybrook Research Institute（桑尼布鲁克研究所）；KTH Royal Institute of Technology（皇家理工学院）
University College London（伦敦大学学院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；MIT（麻省理工学院）；Technical University of Denmark（丹麦技术大学）
Segmentation of structural and diffusion MRI (sMRI/dMRI) is usually performed independently in neuroimaging pipelines. However, some brain structures (e.g., globus pallidus, thalamus and its nuclei) can be extracted more accurately by fusing the two modalities. Following the framework of Bayesian segmentation with probabilistic atlases and unsupervised appearance modeling, we present here a novel algorithm to jointly segment multi-modal sMRI/dMRI data. We propose a hierarchical likelihood term for the dMRI defined on the unit ball, which combines the Beta and Dimroth-Scheidegger-Watson distrib…
University College London（伦敦大学学院）；Kagenova Limited（Kagenova有限公司）
This paper demonstrates spherical convolutional neural networks (S-CNN) offer distinct advantages over conventional fully-connected networks (FCN) at estimating scalar parameters of tissue microstructure from diffusion MRI (dMRI). Such microstructure parameters are valuable for identifying pathology and quantifying its extent. However, current clinical practice commonly acquires dMRI data consisting of only 6 diffusion weighted images (DWIs), limiting the accuracy and precision of estimated microstructure indices. Machine learning (ML) has been proposed to address this challenge. However, exis…
Technical University of Munich（慕尼黑工业大学）；GE Global Research（通用电气全球研发中心）；Harvard Medical School（哈佛医学院）
Perfusion-weighted magnetic resonance imaging (MRI) is an imaging technique that allows one to measure tissue perfusion in an organ of interest through the injection of an intravascular paramagnetic contrast agent (CA). Due to a preference for high temporal and spatial resolution in many applications, this modality could significantly benefit from accelerated data acquisitions. In this paper, we specifically address the problem of reconstructing perfusion MR image series from a subset of k-space data. Our proposed approach is motivated by the observation that temporal variations (dynamics) in …
Deep learning (DL) models are capable of successfully exploiting latent representations in MR data and have become state-of-the-art for accelerated MRI reconstruction. However, undersampling the measurements in k-space as well as the over- or under-parameterized and non-transparent nature of DL make these models exposed to uncertainty. Consequently, uncertainty estimation has become a major issue in DL MRI reconstruction. To estimate uncertainty, Monte Carlo (MC) inference techniques have become a common practice where multiple reconstructions are utilized to compute the variance in reconstruc…
Current deep learning approaches for diffusion MRI modeling circumvent the need for densely-sampled diffusion-weighted images (DWIs) by directly predicting microstructural indices from sparsely-sampled DWIs. However, they implicitly make unrealistic assumptions of static $q$-space sampling during training and reconstruction. Further, such approaches can restrict downstream usage of variably sampled DWIs for usages including the estimation of microstructural indices or tractography. We propose a generative adversarial translation framework for high-quality DWI synthesis with arbitrary $q$-space…
Otto-von-Guericke University（奥托·冯·格里克大学）；University Hospital of Magdeburg（马格德堡大学医院）
Deep learning-based segmentation of the liver and hepatic lesions therein steadily gains relevance in clinical practice due to the increasing incidence of liver cancer each year. Whereas various network variants with overall promising results in the field of medical image segmentation have been successfully developed over the last years, almost all of them struggle with the challenge of accurately segmenting hepatic lesions in magnetic resonance imaging (MRI). This led to the idea of combining elements of convolutional and transformer-based architectures to overcome the existing limitations. T…
