Deep learning-based single image super-resolution (SISR) methods face various challenges when applied to 3D medical volumetric data (i.e., CT and MR images) due to the high memory cost and anisotropic resolution, which adversely affect their performance. Furthermore, mainstream SISR methods are designed to work over specific upsampling factors, which makes them ineffective in clinical practice. In this paper, we introduce a Spatially Aware Interpolation NeTwork (SAINT) for medical slice synthesis to alleviate the memory constraint that volumetric data poses. Compared to other super-resolution methods, SAINT utilizes voxel spacing information to provide desirable levels of details, and allows for the upsampling factor to be determined on the fly. Our evaluations based on 853 CT scans from four datasets that contain liver, colon, hepatic vessels, and kidneys show that SAINT consistently outperforms other SISR methods in terms of medical slice synthesis quality, while using only a single model to deal with different upsampling factors.
Medical image slice interpolation is an active field of research. The methods for this task can be categorized into two broad groups: intensity-based and object-based interpolation methods. While intensity-based methods are generally easier to perform and less computationally expensive, object-based methods are capable of producing more accurate results and account for deformable changes in the objects within the slices. In this paper, performance of two well-known object-based interpolation methods is analyzed and compared. Here, a deformable registration-based method specifically designed fo…
The University of Maryland, College Park（马里兰大学帕克分校）；University of Rochester（罗切斯特大学）；Chinese Academy of Sciences（中国科学院）；Peng Cheng Laboratory（鹏城实验室）
We propose a marginal super-resolution (MSR) approach based on 2D convolutional neural networks (CNNs) for interpolating an anisotropic brain magnetic resonance scan along the highly under-sampled direction, which is assumed to axial without loss of generality. Previous methods for slice interpolation only consider data from pairs of adjacent 2D slices. The possibility of fusing information from the direction orthogonal to the 2D slices remains unexplored. Our approach performs MSR in both sagittal and coronal directions, which provides an initial estimate for slice interpolation. The interpol…
South China University of Technology（华南理工大学）；Rochester Institute of Technology（罗切斯特理工学院）
We introduce a novel frame-interpolation-based method for slice imputation to improve segmentation accuracy for anisotropic 3D medical images, in which the number of slices and their corresponding segmentation labels can be increased between two consecutive slices in anisotropic 3D medical volumes. Unlike previous inter-slice imputation methods, which only focus on the smoothness in the axial direction, this study aims to improve the smoothness of the interpolated 3D medical volumes in all three directions: axial, sagittal, and coronal. The proposed multitask inter-slice imputation method, in …
South China University of Technology（华南理工大学）；University of Utah（犹他大学）
We introduce the idea of inter-slice image augmentation whereby the numbers of the medical images and the corresponding segmentation labels are increased between two consecutive images in order to boost medical image segmentation accuracy. Unlike conventional data augmentation methods in medical imaging, which only increase the number of training samples directly by adding new virtual samples using simple parameterized transformations such as rotation, flipping, scaling, etc., we aim to augment data based on the relationship between two consecutive images, which increases not only the number b…
Adaptive Spatial Transcriptomics Interpolation via Cross-modal Cross-slice Modeling
School of Science and Engineering, University of Dundee, UK（邓迪大学科学与工程学院）；College of Medicine and Biological Information Engineering, Northeastern University, China（东北大学医学与生物信息工程学院）；Department of Clinical Neurosciences, University of Cambridge, UK（剑桥大学临床神经科学系）；School of Medicine, University of Dundee, UK（邓迪大学医学院）；Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK（剑桥大学应用数学与理论物理系）
University of Ottawa（渥太华大学）；the Korea University Anam Hospital（高丽大学安岩医院）
Image interpolation in medical area is of high importance as most 3D biomedical volume images are sampled where the distance between consecutive slices significantly greater than the in-plane pixel size due to radiation dose or scanning time. Image interpolation creates a number of new slices between known slices in order to obtain an isotropic volume image. The results can be used for the higher quality of 3D reconstruction and visualization of human body structures. Semantic interpolation on the manifold has been proved to be very useful for smoothing image interpolation. Nevertheless, all p…
Deep Slice Interpolation for Reducing Through-Plane Anisotropy and Noise in Head CT
Department of Computer Science and Artificial Intelligence, University of Seville（塞维利亚大学计算机科学与人工智能系）；Bioaraba Health Research Institute（Bioaraba健康研究 institute）；IKERBASQUE, Basque Foundation of Science（巴斯克科学基金会）
Amsterdam University Medical Centers（阿姆斯特丹大学医学中心）；University of Amsterdam（阿姆斯特丹大学）；Amsterdam Cardiovascular Sciences（阿姆斯特丹心血管科学研究所）；Informatics Institute, University of Amsterdam（阿姆斯特丹大学信息学研究所）
High-resolution medical images are beneficial for analysis but their acquisition may not always be feasible. Alternatively, high-resolution images can be created from low-resolution acquisitions using conventional upsampling methods, but such methods cannot exploit high-level contextual information contained in the images. Recently, better performing deep-learning based super-resolution methods have been introduced. However, these methods are limited by their supervised character, i.e. they require high-resolution examples for training. Instead, we propose an unsupervised deep learning semanti…
A. K. Choudhury School of Information Technology, University of Calcutta（加尔各答大学A.K.乔杜里信息技术学院）；Indian Statistical Institute（印度统计研究所）
Magnetic Resonance Imaging (MRI) is a technology for non-invasive imaging of anatomical features in detail. It can help in functional analysis of organs of a specimen but it is very costly. In this work, methods for (i) virtual three-dimensional (3D) reconstruction from a single sequence of two-dimensional (2D) slices of MR images of a human spine and brain along a single axis, and (ii) generation of missing inter-slice data are proposed. Our approach helps in preserving the edges, shape, size, as well as the internal tissue structures of the object being captured. The sequence of original 2D …
Xidian University（西安电子科技大学）；Northwestern Polytechnical University（西北工业大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Nankai University（南开大学）；City University of Hong Kong（香港城市大学）
Due to the constraints of the imaging device and high cost in operation time, computer tomography (CT) scans are usually acquired with low intra-slice resolution. Improving the intra-slice resolution is beneficial to the disease diagnosis for both human experts and computer-aided systems. To this end, this paper builds a novel medical slice synthesis to increase the between-slice resolution. Considering that the ground-truth intermediate medical slices are always absent in clinical practice, we introduce the incremental cross-view mutual distillation strategy to accomplish this task in the sel…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Heidelberg University（海德堡大学）
The technical advances in Computed Tomography (CT) allow to obtain immense amounts of 3D data. For such datasets it is very costly and time-consuming to obtain the accurate 3D segmentation markup to train neural networks. The annotation is typically done for a limited number of 2D slices, followed by an interpolation. In this work, we propose a pre-training method SortingLoss. It performs pre-training on slices instead of volumes, so that a model could be fine-tuned on a sparse set of slices, without the interpolation step. Unlike general methods (e.g. SimCLR or Barlow Twins), the task specifi…
Shanghai University（上海大学）；ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）
The thick-slice magnetic resonance (MR) images are often structurally blurred in coronal and sagittal views, which causes harm to diagnosis and image post-processing. Deep learning (DL) has shown great potential to re-construct the high-resolution (HR) thin-slice MR images from those low-resolution (LR) cases, which we refer to as the slice interpolation task in this work. However, since it is generally difficult to sample abundant paired LR-HR MR images, the classical fully supervised DL-based models cannot be effectively trained to get robust performance. To this end, we propose a novel Two-…
Latent Interpolation Learning Using Diffusion Models for Cardiac Volume Reconstruction
School of Computation, Information and Technology, Technical University Munich（技术大学慕尼黑计算信息学院）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Department of Quantitative Biomedicine, University of Zurich（苏黎世大学定量生物医学系）；Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford（牛津大学生物医学工程研究所）；Imperial College London（伦敦帝国学院）；University of Sheffield（谢菲尔德大学）；NVIDIA（英伟达）；Department of Computing, Imperial College London（伦敦帝国学院计算机系）；Department AIBE of Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡大学AIBE系）；School of Medicine, Klinikum Rechts der Isar, Technical University of Munich（慕尼黑技术大学医学院，莱纳特医院）
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（北京邮电大学信息与通信工程学院）；Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）
Deep learning in k-space has demonstrated great potential for image reconstruction from undersampled k-space data in fast magnetic resonance imaging (MRI). However, existing deep learning-based image reconstruction methods typically apply weight-sharing convolutional neural networks (CNNs) to k-space data without taking into consideration the k-space data's spatial frequency properties, leading to ineffective learning of the image reconstruction models. Moreover, complementary information of spatially adjacent slices is often ignored in existing deep learning methods. To overcome such limitati…
Self-Supervised Z-Slice Augmentation for 3D Bio-Imaging via Knowledge Distillation
Collège de France（法兰西学院）；Université PSL（巴黎文理研究大学）；Université Grenoble-Alpes（格勒诺布尔-阿尔卑斯大学）；Sorbonne University（索邦大学）；National Institutes of Health（美国国立卫生研究院）；Hospices Civils de Lyon（里昂公立医院）；Claude Bernard Lyon 1 University（里昂第一大学克洛德·贝尔纳）
Diffusion MRI (dMRI) of the developing brain can provide valuable insights into the white matter development. However, slice thickness in fetal dMRI is typically high (i.e., 3-5 mm) to freeze the in-plane motion, which reduces the sensitivity of the dMRI signal to the underlying anatomy. In this study, we aim at overcoming this problem by using autoencoders to learn unsupervised efficient representations of brain slices in a latent space, using raw dMRI signals and their spherical harmonics (SH) representation. We first learn and quantitatively validate the autoencoders on the developing Human…
ECLARE: Efficient cross-planar learning for anisotropic resolution enhancement
Johns Hopkins University（约翰霍普金斯大学）；Vanderbilt University Medical Center（范德比尔特大学医学中心）；Vanderbilt University Institute of Imaging Science（范德比尔特大学成像科学研究所）；Uniformed Services University（统一服务大学）；Johns Hopkins School of Medicine（约翰霍普金斯医学院）
Quantitative MRI methods that estimate multiple physical parameters simultaneously often require the fitting of a computational complex signal model defined through the Bloch equations. Repeated Bloch simulations can be avoided by matching the measured signal with a precomputed signal dictionary on a discrete parameter grid, as used in MR Fingerprinting. However, accurate estimation requires discretizing each parameter with a high resolution and consequently high computational and memory costs for dictionary generation, storage, and matching. Here, we reduce the required parameter resolution b…
Acquiring High Resolution (HR) Magnetic Resonance (MR) images requires the patient to remain still for long periods of time, which causes patient discomfort and increases the probability of motion induced image artifacts. A possible solution is to acquire low resolution (LR) images and to process them with the Super Resolution Generative Adversarial Network (SRGAN) to create a super-resolved version. This work applies SRGAN to MR images of the prostate and performs three experiments. The first experiment explores improving the in-plane MR image resolution by factors of 4 and 8, and shows that,…
High resolution magnetic resonance~(MR) imaging~(MRI) is desirable in many clinical applications, however, there is a trade-off between resolution, speed of acquisition, and noise. It is common for MR images to have worse through-plane resolution~(slice thickness) than in-plane resolution. In these MRI images, high frequency information in the through-plane direction is not acquired, and cannot be resolved through interpolation. To address this issue, super-resolution methods have been developed to enhance spatial resolution. As an ill-posed problem, state-of-the-art super-resolution methods r…
Machine Learning Derived Blood Input for Dynamic PET Images of Rat Heart
Computer Science, University of Virginia（计算机科学，弗吉尼亚大学）；Radiology, University of Virginia（放射学，弗吉尼亚大学）；Biomedical Engineering, Charlottesville, VA, United States（生物医学工程，弗吉尼亚州夏洛特维尔）
Mayo Clinic（梅奥诊所）；University of Minnesota（明尼苏达大学）；Google LLC（谷歌有限责任公司）
There is a growing demand for high-resolution (HR) medical images in both the clinical and research applications. Image quality is inevitably traded off with the acquisition time for better patient comfort, lower examination costs, dose, and fewer motion-induced artifacts. For many image-based tasks, increasing the apparent resolution in the perpendicular plane to produce multi-planar reformats or 3D images is commonly used. Single image super-resolution (SR) is a promising technique to provide HR images based on unsupervised learning to increase resolution of a 2D image, but there are few rep…
Learning from spatially inhomogenous data: resolution-adaptive convolutions for multiple sclerosis lesion segmentation
University Institute of Diagnostic and Interventional Neuroradiology（诊断与介入神经放射学大学研究所）；Inselspital, Bern University Hospital and University of Bern（伯尔尼大学医院因塞尔spital与伯尔尼大学）；Centre for Artificial Intelligence in Medicine（医学人工智能中心）；University of Bern（伯尔尼大学）
