Multi-Slice Fusion for Sparse-View and Limited-Angle 4D CT Reconstruction
School of Electrical and Computer Engineering, Purdue University（普渡大学电气与计算机工程学院）；Eli Lilly and Company（礼来公司）；Department of Mathematics, Purdue University（普渡大学数学系）
Soumendu Majee, Thilo Balke, Craig A. J. Kemp, Gregery T. Buzzard, Charles A. Bouman
备注：
arXiv admin note: substantial text overlap with arXiv:1906.06601
Inverse problems spanning four or more dimensions such as space, time and other independent parameters have become increasingly important. State-of-the-art 4D reconstruction methods use model based iterative reconstruction (MBIR), but depend critically on the quality of the prior modeling. Recently, plug-and-play (PnP) methods have been shown to be an effective way to incorporate advanced prior models using state-of-the-art denoising algorithms. However, state-of-the-art denoisers such as BM4D and deep convolutional neural networks (CNNs) are primarily available for 2D or 3D images and extending them to higher dimensions is difficult due to algorithmic complexity and the increased difficulty of effective training.
In this paper, we present multi-slice fusion, a novel algorithm for 4D reconstruction, based on the fusion of multiple low-dimensional denoisers. Our approach uses multi-agent consensus equilibrium (MACE), an extension of plug-and-play, as a framework for integrating the multiple lower-dimensional models. We apply our method to 4D cone-beam X-ray CT reconstruction for non destructive evaluation (NDE) of samples that are dynamically moving during acquisition. We implement multi-slice fusion on distributed, heterogeneous clusters in order to reconstruct large 4D volumes in reasonable time and demonstrate the inherent parallelizable nature of the algorithm. We present simulated and real experimental results on sparse-view and limited-angle CT data to demonstrate that multi-slice fusion can substantially improve the quality of reconstructions relative to traditional methods, while also being practical to implement and train.
Zhejiang University（浙江大学）；Shandong Medical Imaging Research Institute（山东医学影像学研究所）；Cheeloo College of Medicine, Shandong University（山东大学齐鲁医学院）；Shanghai Jiao Tong University（上海交通大学）
Multi-slice magnetic resonance images of the fetal brain are usually contaminated by severe and arbitrary fetal and maternal motion. Hence, stable and robust motion correction is necessary to reconstruct high-resolution 3D fetal brain volume for clinical diagnosis and quantitative analysis. However, the conventional registration-based correction has a limited capture range and is insufficient for detecting relatively large motions. Here, we present a novel Affinity Fusion-based Framework for Iteratively Random Motion (AFFIRM) correction of the multi-slice fetal brain MRI. It learns the sequent…
MOSformer: Momentum encoder-based inter-slice fusion transformer for medical image segmentation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Department of Vascular Surgery, Peking Union Medical College Hospital（血管外科部门，北京协和医科大学医院）
The University of Maryland, College Park（马里兰大学帕克分校）；University of Rochester（罗切斯特大学）；Chinese Academy of Sciences（中国科学院）；Peng Cheng Laboratory（鹏城实验室）
We propose a marginal super-resolution (MSR) approach based on 2D convolutional neural networks (CNNs) for interpolating an anisotropic brain magnetic resonance scan along the highly under-sampled direction, which is assumed to axial without loss of generality. Previous methods for slice interpolation only consider data from pairs of adjacent 2D slices. The possibility of fusing information from the direction orthogonal to the 2D slices remains unexplored. Our approach performs MSR in both sagittal and coronal directions, which provides an initial estimate for slice interpolation. The interpol…
Purdue University（普渡大学）；Eli Lilly and Company（礼来公司）
There is an increasing need to reconstruct objects in four or more dimensions corresponding to space, time and other independent parameters. The best 4D reconstruction algorithms use regularized iterative reconstruction approaches such as model based iterative reconstruction (MBIR), which depends critically on the quality of the prior modeling. Recently, Plug-and-Play methods have been shown to be an effective way to incorporate advanced prior models using state-of-the-art denoising algorithms designed to remove additive white Gaussian noise (AWGN). However, state-of-the-art denoising algorith…
Deepwise AI Lab（深睿人工智能实验室）；The University of Hong Kong（香港大学）；Sun Yat-sen University（中山大学）
Recently 3D volumetric organ segmentation attracts much research interest in medical image analysis due to its significance in computer aided diagnosis. This paper aims to address the pancreas segmentation task in 3D computed tomography volumes. We propose a novel end-to-end network, Globally Guided Progressive Fusion Network, as an effective and efficient solution to volumetric segmentation, which involves both global features and complicated 3D geometric information. A progressive fusion network is devised to extract 3D information from a moderate number of neighboring slices and predict a p…
Improved 3D Whole Heart Geometry from Sparse CMR Slices
School of Biomedical Engineering and Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与影像科学学院）；College of Mathematical Medicine, Zhejiang Normal University（浙江师范大学数学医学院）；Imperial College London（伦敦帝国学院）；HeartFlow, Inc.（HeartFlow公司）；University/BHF Centre for Cardiovascular Science, University of Edinburgh（爱丁堡大学大学/英国心脏基金会心血管科学中心）
Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局资讯通信研究院）；National University of Singapore（新加坡国立大学）；Singapore General Hospital（新加坡中央医院）；Duke-NUS Medical School Singapore（新加坡杜克-新加坡国立大学医学院）
Accurate automatic liver and tumor segmentation plays a vital role in treatment planning and disease monitoring. Recently, deep convolutional neural network (DCNNs) has obtained tremendous success in 2D and 3D medical image segmentation. However, 2D DCNNs cannot fully leverage the inter-slice information, while 3D DCNNs are computationally expensive and memory intensive. To address these issues, we first propose a novel dense-sparse training flow from a data perspective, in which, densely adjacent slices and sparsely adjacent slices are extracted as inputs for regularizing DCNNs, thereby impro…
Xidian University（西安电子科技大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
3D image segmentation is one of the most important and ubiquitous problems in medical image processing. It provides detailed quantitative analysis for accurate disease diagnosis, abnormal detection, and classification. Currently deep learning algorithms are widely used in medical image segmentation, most algorithms trained models with full annotated datasets. However, obtaining medical image datasets is very difficult and expensive, and full annotation of 3D medical image is a monotonous and time-consuming work. Partially labelling informative slices in 3D images will be a great relief of manu…
Image fusion is a significant problem in many fields including digital photography, computational imaging and remote sensing, to name but a few. Recently, deep learning has emerged as an important tool for image fusion. This paper presents three deep convolutional sparse coding (CSC) networks for three kinds of image fusion tasks (i.e., infrared and visible image fusion, multi-exposure image fusion, and multi-modal image fusion). The CSC model and the iterative shrinkage and thresholding algorithm are generalized into dictionary convolution units. As a result, all hyper-parameters are learned …
Modeling 3D context is essential for high-performance 3D medical image analysis. Although 2D networks benefit from large-scale 2D supervised pretraining, it is weak in capturing 3D context. 3D networks are strong in 3D context yet lack supervised pretraining. As an emerging technique, \emph{3D context fusion operator}, which enables conversion from 2D pretrained networks, leverages the advantages of both and has achieved great success. Existing 3D context fusion operators are designed to be spatially symmetric, i.e., performing identical operations on each 2D slice like convolutions. However, …
Recent advances in transformer-based models have drawn attention to exploring these techniques in medical image segmentation, especially in conjunction with the U-Net model (or its variants), which has shown great success in medical image segmentation, under both 2D and 3D settings. Current 2D based methods either directly replace convolutional layers with pure transformers or consider a transformer as an additional intermediate encoder between the encoder and decoder of U-Net. However, these approaches only consider the attention encoding within one single slice and do not utilize the axial-a…
Vision-Language Model Based Multi-Expert Fusion for CT Image Classification
College of Computer Science and Artificial Intelligence, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University（复旦大学计算机科学与人工智能学院，上海智能信息处理重点实验室）；University of Oxford（牛津大学）；The Hong Kong University of Science and Technology（香港科技大学）
Biomedical Image Analysis Group, Imperial College London, UK（帝国理工学院伦敦生物医学图像分析组）
Real-world settings often do not allow acquisition of high-resolution volumetric images for accurate morphological assessment and diagnostic. In clinical practice it is frequently common to acquire only sparse data (e.g. individual slices) for initial diagnostic decision making. Thereby, physicians rely on their prior knowledge (or mental maps) of the human anatomy to extrapolate the underlying 3D information. Accurate mental maps require years of anatomy training, which in the first instance relies on normative learning, i.e. excluding pathology. In this paper, we leverage Bayesian Deep Learn…
ShanghaiTech University（上海科技大学）；Chinese Academy of Sciences（中国科学院）；Shanghai Institute of Microsystem and Information Technology（上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Photoacoustic (PA) imaging is a biomedical imaging modality capable of acquiring high-contrast images of optical absorption at depths much greater than traditional optical imaging techniques. However, practical instrumentation and geometry limit the number of available acoustic sensors surrounding the imaging target, which results in the sparsity of sensor data. Conventional PA image reconstruction methods give severe artifacts when they are applied directly to the sparse PA data. In this paper, we firstly propose to employ a novel signal processing method to make sparse PA raw data more suita…
Nanjing Institute of Intelligent Technology（南京智能技术研究院）
Event camera is a novel bio-inspired vision sensor that outputs event stream. In this paper, we propose a novel data fusion algorithm called EAS to fuse conventional intensity images with the event stream. The fusion result is applied to some ego-motion estimation frameworks, and is evaluated on a public dataset acquired in dim scenes. In our 3-DoF rotation estimation framework, EAS achieves the highest estimation accuracy among intensity images and representations of events including event slice, TS and SITS. Compared with original images, EAS reduces the average APE by 69%, benefiting from t…
Sparse-view computed tomography (CT) is known as a widely used approach to reduce radiation dose while accelerating imaging through lowered projection views and correlated calculations. However, its severe imaging noise and streaking artifacts turn out to be a major issue in the low dose protocol. In this paper, we propose a dual-domain deep learning-based method that breaks through the limitations of currently prevailing algorithms that merely process single image slices. Since the scanned object usually contains a high degree of spatial continuity, the obtained consecutive imaging slices emb…
Conditional Diffusion Posterior Alignment for Sparse-View CT Reconstruction
Swiss Data Science Center (SDSC) in Paul Scherrer Institute (PSI)（瑞士数据科学中心（SDSC）在保罗·舍勒研究所（PSI））；Swiss Data Science Center (SDSC) and ETH Zurich（瑞士数据科学中心（SDSC）和苏黎世联邦理工学院）；Swiss Data Science Center (SDSC) and Paul Scherrer Institute (PSI)（瑞士数据科学中心（SDSC）和保罗·舍勒研究所（PSI））
CT imaging works by reconstructing an object of interest from a collection of projections. Traditional methods such as filtered-back projection (FBP) work on projection images acquired around a fixed rotation axis. However, for some CT problems, it is desirable to perform a joint reconstruction from projection data acquired from multiple rotation axes. In this paper, we present Multi-Pose Fusion, a novel algorithm that performs a joint tomographic reconstruction from CT scans acquired from multiple poses of a single object, where each pose has a distinct rotation axis. Our approach uses multi-…
Harbin Institute of Technology（哈尔滨工业大学）；Minjiang University（闽江学院）；Nanjing University of Science and Technology（南京理工大学）
Sparse Representation (SR) techniques encode the test samples into a sparse linear combination of all training samples and then classify the test samples into the class with the minimum residual. The classification of SR techniques depends on the representation capability on the test samples. However, most of these models view the representation problem of the test samples as a deterministic problem, ignoring the uncertainty of the representation. The uncertainty is caused by two factors, random noise in the samples and the intrinsic randomness of the sample set, which means that if we capture…
Optimization in Sparse 2D to Dense 3D Weakly Supervised Learning: Application to Multi-Label Segmentation of Large ex vivo MRI Data
NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montreal（神经多极实验室，生物医学工程学院，蒙特利尔理工学院）；École Centrale de Lyon（里昂中央理工学院）；Mila - Quebec AI Institute（魁北克人工智能研究所）；Functional Neuroimaging Unit, CRIUGM, University of Montreal（功能神经影像单元，CRIUGM，蒙特利尔大学）；Translational Neuroradiology Section, National Institute of Neurological Disorders and Stroke, National Institutes of Health（转化神经放射学部门，国家神经疾病与中风研究所，国家卫生研究院）；Translational Imaging in Neurology (ThINk) Basel, Department of Biomedical Engineering, Faculty of Medicine, University Hospital Basel and University of Basel（神经学转化成像（ThINk）巴塞尔，生物医学工程系，医学院，巴塞尔大学医院和巴塞尔大学）；Neurologic Clinic and Policlinic, Departments of Medicine, University Hospital Basel, Switzerland（神经科诊所和多科诊所，医学院，巴塞尔大学医院，瑞士）；Research Center for Clinical Neuroimmunology and Neuroscience Basel (RC2NB), University Hospital Basel and University of Basel, Switzerland（临床神经免疫学和神经科学巴塞尔研究中心（RC2NB），巴塞尔大学医院和巴塞尔大学，瑞士）；National Institute of Neurological Disorders and Stroke, National Institutes of Health（国家神经疾病与中风研究所，国家卫生研究院）；Centre de recherche du CHU Sainte-Justine, Université de Montréal, Montreal, QC, Canada（圣朱斯特医院研究中心，蒙特利尔大学，蒙特利尔，魁北克，加拿大）；Quantitative MRI core facility, NINDS, NIH（定量MRI核心设施，NINDS，NIH）；Experimental Immunotherapeutics Unit, Division of Neuroimmunology and Neurovirology, NINDS, NIH（实验免疫治疗单元，神经免疫学和神经病毒学部门，NINDS，NIH）
In this paper, the Primal-Dual UNet for sparse view CT reconstruction is modified to be applicable to cone beam projections and perform reconstructions of entire volumes instead of slices. Experiments show that the PSNR of the proposed method is increased by 10dB compared to the direct FDK reconstruction and almost 3dB compared to the modified original Primal-Dual Network when using only 23 projections. The presented network is not optimized wrt. memory consumption or hyperparameters but merely serves as a proof of concept and is limited to low resolution projections and volumes.
