Deep Sequential Segmentation of Organs in Volumetric Medical Scans
Alexey Novikov, David Major, Maria Wimmer, Dimitrios Lenis, Katja Bühler
发表信息：
Published in IEEE Transactions on Medical Imaging on 16 November 2018, URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8537944&isnumber=4359023
Segmentation in 3D scans is playing an increasingly important role in current clinical practice supporting diagnosis, tissue quantification, or treatment planning. The current 3D approaches based on convolutional neural networks usually suffer from at least three main issues caused predominantly by implementation constraints - first, they require resizing the volume to the lower-resolutional reference dimensions, second, the capacity of such approaches is very limited due to memory restrictions, and third, all slices of volumes have to be available at any given training or testing time. We address these problems by a U-Net-like architecture consisting of bidirectional convolutional LSTM and convolutional, pooling, upsampling and concatenation layers enclosed into time-distributed wrappers. Our network can either process the full volumes in a sequential manner, or segment slabs of slices on demand. We demonstrate performance of our architecture on vertebrae and liver segmentation tasks in 3D CT scans.
BOWEI INTEGRATED CIRCUITS CO.,LTD.（博威集成电路有限公司）；The 13th Research Institute of China Electronics Technology Group Corporation（中国电子科技集团公司第十三研究所）；Jilin University（吉林大学）
The present study proposes a novel 3D display contains a micro-volumetric scanning system (MVS) and a real time reconstruction hologram system (RTRH).
The Johns Hopkins University（约翰斯·霍普金斯大学）；The Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）
There has been a debate on whether to use 2D or 3D deep neural networks for volumetric organ segmentation. Both 2D and 3D models have their advantages and disadvantages. In this paper, we present an alternative framework, which trains 2D networks on different viewpoints for segmentation, and builds a 3D Volumetric Fusion Net (VFN) to fuse the 2D segmentation results. VFN is relatively shallow and contains much fewer parameters than most 3D networks, making our framework more efficient at integrating 3D information for segmentation. We train and test the segmentation and fusion modules individu…
CT-ScanGaze: A Dataset and Baselines for 3D Volumetric Scanpath Modeling
University of Arkansas（亚拉巴马大学）；University of Houston（休斯顿大学）；University of Science VNU-HCM（越南国家大学VNU-HCM）；FPT Software（FPT软件公司）；Auburn University（阿伯丁大学）；University of Liverpool（利物浦大学）；MD Anderson Cancer Center（MD安德森癌症中心）
LMU München（慕尼黑大学）；Technische Universität München（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Fully Convolutional Neural Networks (F-CNNs) achieve state-of-the-art performance for image segmentation in medical imaging. Recently, squeeze and excitation (SE) modules and variations thereof have been introduced to recalibrate feature maps channel- and spatial-wise, which can boost performance while only minimally increasing model complexity. So far, the development of SE has focused on 2D images. In this paper, we propose `Project & Excite' (PE) modules that base upon the ideas of SE and extend them to operating on 3D volumetric images. `Project & Excite' does not perform global average po…
LMU München（慕尼黑大学）；Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Deep neural networks enable highly accurate image segmentation, but require large amounts of manually annotated data for supervised training. Few-shot learning aims to address this shortcoming by learning a new class from a few annotated support examples. We introduce, a novel few-shot framework, for the segmentation of volumetric medical images with only a few annotated slices. Compared to other related works in computer vision, the major challenges are the absence of pre-trained networks and the volumetric nature of medical scans. We address these challenges by proposing a new architecture f…
Technical University of Munich（慕尼黑工业大学）；Ulm University（乌尔姆大学）
The lack of sufficient annotated image data is a common issue in medical image segmentation. For some organs and densities, the annotation may be scarce, leading to poor model training convergence, while other organs have plenty of annotated data. In this work, we present MetaMedSeg, a gradient-based meta-learning algorithm that redefines the meta-learning task for the volumetric medical data with the goal to capture the variety between the slices. We also explore different weighting schemes for gradients aggregation, arguing that different tasks might have different complexity, and hence, con…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
CNN-based volumetric methods that label individual voxels now dominate the field of biomedical segmentation. However, 3D surface representations are often required for proper analysis. They can be obtained by post-processing the labeled volumes which typically introduces artifacts and prevents end-to-end training. In this paper, we therefore introduce a novel architecture that goes directly from 3D image volumes to 3D surfaces without post-processing and with better accuracy than current methods. We evaluate it on Electron Microscopy and MRI brain images as well as CT liver scans. We will show…
Magnetic resonance imaging (MRI) enables 3-D imaging of anatomical structures. However, the acquisition of MR volumes with high spatial resolution leads to long scan times. To this end, we propose volumetric super-resolution forests (VSRF) to enhance MRI resolution retrospectively. Our method learns a locally linear mapping between low-resolution and high-resolution volumetric image patches by employing random forest regression. We customize features suitable for volumetric MRI to train the random forest and propose a median tree ensemble for robust regression. VSRF outperforms state-of-the-ar…
Stanford University（斯坦福大学）；Technical University of Munich（慕尼黑工业大学）
We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network. In contrast to existing methods that either use geometry or RGB data as input for this task, we combine both data modalities in a joint, end-to-end network architecture. Rather than simply projecting color data into a volumetric grid and operating solely in 3D -- which would result in insufficient detail -- we first extract feature maps from associated RGB images. These features are then mapped into the volumetric feature grid of a 3D network …
University of Brescia（布雷西亚大学）；University of Glasgow（格拉斯哥大学）
Many functional and structural neuroimaging studies call for accurate morphometric segmentation of different brain structures starting from image intensity values of MRI scans. Current automatic (multi-) atlas-based segmentation strategies often lack accuracy on difficult-to-segment brain structures and, since these methods rely on atlas-to-scan alignment, they may take long processing times. Recently, methods deploying solutions based on Convolutional Neural Networks (CNNs) are making the direct analysis of out-of-the-scanner data feasible. However, current CNN-based solutions partition the t…
Infervision Medical Technology Co., Ltd.（推想医疗科技有限公司）；UCLA（加州大学洛杉矶分校）
In clinical practice, anisotropic volumetric medical images with low through-plane resolution are commonly used due to short acquisition time and lower storage cost. Nevertheless, the coarse resolution may lead to difficulties in medical diagnosis by either physicians or computer-aided diagnosis algorithms. Deep learning-based volumetric super-resolution (SR) methods are feasible ways to improve resolution, with convolutional neural networks (CNN) at their core. Despite recent progress, these methods are limited by inherent properties of convolution operators, which ignore content relevance an…
We present 3DVNet, a novel multi-view stereo (MVS) depth-prediction method that combines the advantages of previous depth-based and volumetric MVS approaches. Our key idea is the use of a 3D scene-modeling network that iteratively updates a set of coarse depth predictions, resulting in highly accurate predictions which agree on the underlying scene geometry. Unlike existing depth-prediction techniques, our method uses a volumetric 3D convolutional neural network (CNN) that operates in world space on all depth maps jointly. The network can therefore learn meaningful scene-level priors. Furtherm…
College of Biomedical Engineering, Rangsit University（兰实大学生物医学工程学院）
Automated medical image segmentation is a priority research area for computational methods. In particular, detection of cancerous tumors represents a current challenge in this area with potential for real-world impact. This paper describes a method developed in response to the 2019 Kidney Tumor Segmentation Challenge (KiTS19). Axial computed tomography (CT) scans from 210 kidney cancer patients were used to develop and evaluate this automatic segmentation method based on a logical ensemble of fully-convolutional network (FCN) architectures, followed by volumetric validation. Data was pre-proce…
ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China（生物医学工程学院，生命科学与医学系，中国科学技术大学）；Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE) Lab, YRD-RIGHT, USTC Suzhou Institute for Advanced Research（医学影像、机器人、分析计算与学习（MIRACLE）实验室，YRD-RIGHT，中国科学技术大学苏州研究院）；Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology（江苏省多模态数字孪生技术重点实验室）；Biomedical Basic Research Center (BBRC) of Jiangsu Province（江苏省生物医学基础研究中心）；Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, USTC（放射科，中国科学技术大学第一附属医院，生命科学与医学系，中国科学技术大学）；Anhui IFLYTEK CO., Ltd（安徽科大讯飞股份有限公司）；School of Medicine, Stanford University（医学院，斯坦福大学）；State Key Laboratory of Precision and Intelligent Chemistry, Hefei, Anhui, China（安徽省精密与智能化学重点实验室，合肥，安徽，中国）
Volume Graphics GmbH（Volume Graphics有限公司）；University Medical Center Mannheim, Heidelberg University（海德堡大学曼海姆大学医学中心）；Heidelberg University（海德堡大学）；University of Padova（帕多瓦大学）
We present the first attempt to perform short glass fiber semantic segmentation from X-ray computed tomography volumetric datasets at medium (3.9 μm isotropic) and low (8.3 μm isotropic) resolution using deep learning architectures. We performed experiments on both synthetic and real CT scans and evaluated deep fully convolutional architectures with both 2D and 3D kernels. Our artificial neural networks outperform existing methods at both medium and low resolution scans.
VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine
MedVisAI Lab Department of ECE Northwestern University（MedVisAI实验室 电子工程系 西北大学）；Institute for Infocomm Research (I 2 R) A*STAR, Singapore（信息与通信研究所（I 2 R）A*STAR，新加坡）；MedVisAI Lab Lee Kong Chian School of Medicine, Nanyang Technological University（MedVisAI实验室 李科田医学院，南洋理工大学）
Khulna University of Engineering & Technology（库尔纳工程技术大学）；University Health Network（大学健康网络）；University of Toronto（多伦多大学）
The novel COVID-19 is a global pandemic disease overgrowing worldwide. Computer-aided screening tools with greater sensitivity is imperative for disease diagnosis and prognosis as early as possible. It also can be a helpful tool in triage for testing and clinical supervision of COVID-19 patients. However, designing such an automated tool from non-invasive radiographic images is challenging as many manually annotated datasets are not publicly available yet, which is the essential core requirement of supervised learning schemes. This article proposes a 3D Convolutional Neural Network (CNN)-based…
Chest X-ray (CXR) is one of the most commonly prescribed medical imaging procedures, often with over 2-10x more scans than other imaging modalities such as MRI, CT scan, and PET scans. These voluminous CXR scans place significant workloads on radiologists and medical practitioners. Organ segmentation is a crucial step to obtain effective computer-aided detection on CXR. In this work, we propose Structure Correcting Adversarial Network (SCAN) to segment lung fields and the heart in CXR images. SCAN incorporates a critic network to impose on the convolutional segmentation network the structural …
Brain cancer can be very fatal, but chances of survival increase through early detection and treatment. Doctors use Magnetic Resonance Imaging (MRI) to detect and locate tumors in the brain, and very carefully analyze scans to segment brain tumors. Manual segmentation is time consuming and tiring for doctors, and it can be difficult for them to notice extremely small abnormalities. Automated segmentations performed by computers offer quicker diagnoses, the ability to notice small details, and more accurate segmentations. Advances in deep learning and computer hardware have allowed for high-per…
University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Max-Planck-Institute for Informatics（马克斯·普朗克信息学研究所）；Stanford University（斯坦福大学）
We present a novel approach for the reconstruction of dynamic geometric shapes using a single hand-held consumer-grade RGB-D sensor at real-time rates. Our method does not require a pre-defined shape template to start with and builds up the scene model from scratch during the scanning process. Geometry and motion are parameterized in a unified manner by a volumetric representation that encodes a distance field of the surface geometry as well as the non-rigid space deformation. Motion tracking is based on a set of extracted sparse color features in combination with a dense depth-based constrain…
Casey Eye Institute, Oregon Health & Science University（俄勒冈健康与科学大学凯西眼科研究所）；School of Physics and Optoelectronic Engineering, Foshan University（佛山大学物理与光电工程学院）；Department of Biomedical Engineering, Oregon Health & Science University（俄勒冈健康与科学大学生物医学工程系）
Purpose: We proposed a deep convolutional neural network (CNN), named Retinal Fluid Segmentation Network (ReF-Net) to segment volumetric retinal fluid on optical coherence tomography (OCT) volume. Methods: 3 x 3-mm OCT scans were acquired on one eye by a 70-kHz OCT commercial AngioVue system (RTVue-XR; Optovue, Inc.) from 51 participants in a clinical diabetic retinopathy (DR) study (45 with retinal edema and 6 healthy controls). A CNN with U-Net-like architecture was constructed to detect and segment the retinal fluid. Cross-sectional OCT and angiography (OCTA) scans were used for training an…
SegQC: a segmentation network-based framework for multi-metric segmentation quality control and segmentation error detection in volumetric medical images
University of Haifa（海法大学）；Tel Aviv Sourasky Medical Center（特拉维夫苏拉斯基医疗中心）；Sackler Faculty of Medicine, Tel Aviv University（特拉维夫大学萨克勒医学院）；Tel Aviv University（特拉维夫大学）；Sagol Brain Institute（萨戈尔脑研究所）；Sagol School of Neuroscience, Tel Aviv University（特拉维夫大学萨戈尔神经科学学院）；School of Computer Science and Engineering, The Hebrew University of Jerusalem（希伯来大学计算机科学与工程学院）；The Hebrew University of Jerusalem（希伯来大学）；Edmond and Lily Safra Center for Brain Sciences, The Hebrew University of Jerusalem（希伯来大学埃德蒙与莉莉·萨夫拉脑科学中心）
Volumetric imaging of samples using fluorescence microscopy plays an important role in various fields including physical, medical and life sciences. Here we report a deep learning-based volumetric image inference framework that uses 2D images that are sparsely captured by a standard wide-field fluorescence microscope at arbitrary axial positions within the sample volume. Through a recurrent convolutional neural network, which we term as Recurrent-MZ, 2D fluorescence information from a few axial planes within the sample is explicitly incorporated to digitally reconstruct the sample volume over …
Deep-Motion-Net: GNN-based volumetric organ shape reconstruction from single-view 2D projections
University of Leeds（利兹大学）；St James’s University Hospital, Leeds Teaching Hospitals NHS Trust（利兹教学医院NHS信托基金圣詹姆斯大学医院）；University of Manchester（曼彻斯特大学）；University of Sheffield（谢菲尔德大学）；University of Glasgow（格拉斯哥大学）
Deep learning algorithms, in particular 2D and 3D fully convolutional neural networks (FCNs), have rapidly become the mainstream methodology for volumetric medical image segmentation. However, 2D convolutions cannot fully leverage the rich spatial information along the third axis, while 3D convolutions suffer from the demanding computation and high GPU memory consumption. In this paper, we propose to automatically search the network architecture tailoring to volumetric medical image segmentation problem. Concretely, we formulate the structure learning as differentiable neural architecture sear…
Case Western Reserve University（凯斯西储大学）；IBM Almaden Research Center（IBM阿尔马登研究中心）
We explore a solution for learning disease signatures from weakly, yet easily obtainable, annotated volumetric medical imaging data by analyzing 3D volumes as a sequence of 2D images. We demonstrate the performance of our solution in the detection of emphysema in lung cancer screening low-dose CT images. Our approach utilizes convolutional long short-term memory (LSTM) to "scan" sequentially through an imaging volume for the presence of disease in a portion of scanned region. This framework allowed effective learning given only volumetric images and binary disease labels, thus enabling trainin…
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
Weakly supervised segmentation is an important problem in medical image analysis due to the high cost of pixelwise annotation. Prior methods, while often focusing on weak labels of 2D images, exploit few structural cues of volumetric medical images. To address this, we propose a novel weakly-supervised segmentation strategy capable of better capturing 3D shape prior in both model prediction and learning. Our main idea is to extract a self-taught shape representation by leveraging weak labels, and then integrate this representation into segmentation prediction for shape refinement. To this end,…
Xiamen University（厦门大学）；The University of Hong Kong（香港大学）；Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）深圳大数据研究院）
Transformer, the model of choice for natural language processing, has drawn scant attention from the medical imaging community. Given the ability to exploit long-term dependencies, transformers are promising to help atypical convolutional neural networks to overcome their inherent shortcomings of spatial inductive bias. However, most of recently proposed transformer-based segmentation approaches simply treated transformers as assisted modules to help encode global context into convolutional representations. To address this issue, we introduce nnFormer, a 3D transformer for volumetric medical i…
Department of Computer Science, Princeton University（普林斯顿大学计算机科学系）；Siemens Healthineers, Digital Services, Digital Technology & Innovation（西门子医疗保健公司数字服务部数字技术与创新）
Automatic delineation and measurement of main organs such as liver is one of the critical steps for assessment of hepatic diseases, planning and postoperative or treatment follow-up. However, addressing this problem typically requires performing computed tomography (CT) scanning and complicated postprocessing of the resulting scans using slice-by-slice techniques. In this paper, we show that 3D organ shape can be automatically predicted directly from topogram images, which are easier to acquire and have limited exposure to radiation during acquisition, compared to CT scans. We evaluate our app…
SurgFormer: Scalable Learning of Organ Deformation with Resection Support and Real-Time Inference
Department of Computer Science, College of Connected Computing, Vanderbilt University（计算机科学系，连接计算学院，范德比尔特大学）；Department of Biomedical Engineering, Vanderbilt University（生物医学工程系，范德比尔特大学）；Department of Mechanical Engineering, Vanderbilt University（机械工程系，范德比尔特大学）；Vanderbilt Institute for Surgery and Engineering（范德比尔特手术与工程研究院）
PAII Inc.（PAII 公司）；Chang Gung Memorial Hospital（长庚纪念医院）；Ping An Technology Co., Ltd.（平安科技（深圳）有限公司）
Identifying, measuring and reporting lesions accurately and comprehensively from patient CT scans are important yet time-consuming procedures for physicians. Computer-aided lesion/significant-findings detection techniques are at the core of medical imaging, which remain very challenging due to the tremendously large variability of lesion appearance, location and size distributions in 3D imaging. In this work, we propose a novel deep anchor-free one-stage VULD framework that incorporates (1) P3DC operators to recycle the architectural configurations and pre-trained weights from the off-the-shel…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Southeast University（东南大学）；Lenovo Research（联想研究院）
Automated segmentation of kidney and tumor from 3D CT scans is necessary for the diagnosis, monitoring, and treatment planning of the disease. In this paper, we describe a two-stage framework for kidney and tumor segmentation based on 3D fully convolutional network (FCN). The first stage preliminarily locate the kidney and cut off the irrelevant background to reduce class imbalance and computation cost. Then the second stage precisely segment the kidney and tumor on the cropped patch. The proposed method ranks the 4th place out of 105 competitive teams in MICCAI 2019 KiTS Challenge with a Comp…
CitAI Research Centre（CitAI研究中心）；City, University of London（伦敦城市大学）；BioMedIA（生物医学影像分析组）；Imperial College（帝国理工学院）
U-Net has been the go-to architecture for medical image segmentation tasks, however computational challenges arise when extending the U-Net architecture to 3D images. We propose the Implicit U-Net architecture that adapts the efficient Implicit Representation paradigm to supervised image segmentation tasks. By combining a convolutional feature extractor with an implicit localization network, our implicit U-Net has 40% less parameters than the equivalent U-Net. Moreover, we propose training and inference procedures to capitalize sparse predictions. When comparing to an equivalent fully convolut…
University College London（伦敦大学学院）；University of Washington（华盛顿大学）
Neuroimaging to neuropathology correlation (NTNC) promises to enable the transfer of microscopic signatures of pathology to in vivo imaging with MRI, ultimately enhancing clinical care. NTNC traditionally requires a volumetric MRI scan, acquired either ex vivo or a short time prior to death. Unfortunately, ex vivo MRI is difficult and costly, and recent premortem scans of sufficient quality are seldom available. To bridge this gap, we present methodology to 3D reconstruct and segment full brain image volumes from brain dissection photographs, which are routinely acquired at many brain banks an…
VAMOS-OCTA: Vessel-Aware Multi-Axis Orthogonal Supervision for Inpainting Motion-Corrupted OCT Angiography Volumes
Department of Computer Science, Vanderbilt University（计算机科学系，范德比尔特大学）；Digital Technology and Innovation, Siemens Healthineers（数字技术与创新，西门子医疗）；Department of Biomedical Engineering, Vanderbilt University（生物医学工程系，范德比尔特大学）
Intel Visual Computing Institute（英特尔视觉计算研究所）；Intel（英特尔公司）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
With the advent of affordable depth sensors, 3D capture becomes more and more ubiquitous and already has made its way into commercial products. Yet, capturing the geometry or complete shapes of everyday objects using scanning devices (e.g. Kinect) still comes with several challenges that result in noise or even incomplete shapes. Recent success in deep learning has shown how to learn complex shape distributions in a data-driven way from large scale 3D CAD Model collections and to utilize them for 3D processing on volumetric representations and thereby circumventing problems of topology and tes…
Fast refacing of MR images with a generative neural network lowers re-identification risk and preserves volumetric consistency
Lausanne University Hospital (CHUV)（洛桑大学医院）；Siemens Healthineers International AG（西门子医疗国际股份公司）；Ecole Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；University of Lausanne (UNIL)（洛桑大学）
Shanghai Jiao Tong University（上海交通大学）；University of Sydney（悉尼大学）
Dynamic medical imaging is usually limited in application due to the large radiation doses and longer image scanning and reconstruction times. Existing methods attempt to reduce the dynamic sequence by interpolating the volumes between the acquired image volumes. However, these methods are limited to either 2D images and/or are unable to support large variations in the motion between the image volume sequences. In this paper, we present a spatiotemporal volumetric interpolation network (SVIN) designed for 4D dynamic medical images. SVIN introduces dual networks: first is the spatiotemporal mot…
Tel-Aviv University（特拉维夫大学）；Sheba Medical Center（舍巴医疗中心）
Pulmonary embolus (PE) refers to obstruction of pulmonary arteries by blood clots. PE accounts for approximately 100,000 deaths per year in the United States alone. The clinical presentation of PE is often nonspecific, making the diagnosis challenging. Thus, rapid and accurate risk stratification is of paramount importance. High-risk PE is caused by right ventricular (RV) dysfunction from acute pressure overload, which in return can help identify which patients require more aggressive therapy. Reconstructed four-chamber views of the heart on chest CT can detect right ventricular enlargement. C…
Senckenberg Biodiversity and Climate Research Centre (SBiK-F)（森肯伯格生物多样性与气候研究中心）；Philipps-University Marburg（马尔堡菲利普斯大学）；Palmengarten der Stadt Frankfurt（法兰克福棕榈园）；Senckenberg Research Institute and Natural History Museum（森肯伯格研究所与自然历史博物馆）
As herbarium specimens are increasingly becoming digitized and accessible in online repositories, advanced computer vision techniques are being used to extract information from them. The presence of certain plant organs on herbarium sheets is useful information in various scientific contexts and automatic recognition of these organs will help mobilize such information. In our study we use deep learning to detect plant organs on digitized herbarium specimens with Faster R-CNN. For our experiment we manually annotated hundreds of herbarium scans with thousands of bounding boxes for six types of …
