The digital factory provides undoubtedly a great potential for future production systems in terms of efficiency and effectivity. A key aspect on the way to realize the digital copy of a real factory is the understanding of complex indoor environments on the basis of 3D data. In order to generate an accurate factory model including the major components, i.e. building parts, product assets and process details, the 3D data collected during digitalization can be processed with advanced methods of deep learning. In this work, we propose a fully Bayesian and an approximate Bayesian neural network for point cloud segmentation. This allows us to analyze how different ways of estimating uncertainty in these networks improve segmentation results on raw 3D point clouds. We achieve superior model performance for both, the Bayesian and the approximate Bayesian model compared to the frequentist one. This performance difference becomes even more striking when incorporating the networks' uncertainty in their predictions. For evaluation we use the scientific data set S3DIS as well as a data set, which was collected by the authors at a German automotive production plant. The methods proposed in this work lead to more accurate segmentation results and the incorporation of uncertainty information makes this approach especially applicable to safety critical applications.
BMW Group（宝马集团）；Alpen-Adria-University Klagenfurt（克拉根福阿尔卑斯-亚德里亚大学）
The 3D modelling of indoor environments and the generation of process simulations play an important role in factory and assembly planning. In brownfield planning cases existing data are often outdated and incomplete especially for older plants, which were mostly planned in 2D. Thus, current environment models cannot be generated directly on the basis of existing data and a holistic approach on how to build such a factory model in a highly automated fashion is mostly non-existent. Major steps in generating an environment model in a production plant include data collection and pre-processing, ob…
Artificial Intelligence in Breast Cancer Care: Transforming Preoperative Planning and Patient Education with 3D Reconstruction
Surgical Artificial Intelligence Research Academy（外科人工智能研究学院）；University Health Network（大学健康网络）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Princess Margaret Cancer Centre（朱莉安娜女王癌症中心）
Mask Factory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation
State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Linköping University（林雪平大学）；VCIP&CS, Nankai University（南开大学VCIP&CS）
NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；MIT（麻省理工学院）；University of Waterloo（滑铁卢大学）
We introduce DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring minimal human effort. Current deep networks are extremely data-hungry, benefiting from training on large-scale datasets, which are time consuming to annotate. Our method relies on the power of recent GANs to generate realistic images. We show how the GAN latent code can be decoded to produce a semantic segmentation of the image. Training the decoder only needs a few labeled examples to generalize to the rest of the latent space, resulting in an infinite annotated…
Amsterdam University Medical Center, University of Amsterdam（阿姆斯特丹大学阿姆斯特丹医学中心）；University Medical Center Utrecht（乌得勒支大学医学中心）；Image Sciences Institute, University Medical Center Utrecht（乌得勒支大学医学中心影像科学研究所）
Convolutional neural networks (CNNs) have been widely and successfully used for medical image segmentation. However, CNNs are typically considered to require large numbers of dedicated expert-segmented training volumes, which may be limiting in practice. This work investigates whether clinically obtained segmentations which are readily available in picture archiving and communication systems (PACS) could provide a possible source of data to train a CNN for segmentation of organs-at-risk (OARs) in radiotherapy treatment planning. In such data, delineations of structures deemed irrelevant to the…
Center for Spatial Information Science, University of Tokyo（东京大学空间信息科学研究中心）；Microbase Inc.（Microbase公司）
The automatic digitizing of paper maps is a significant and challenging task for both academia and industry. As an important procedure of map digitizing, the semantic segmentation section mainly relies on manual visual interpretation with low efficiency. In this study, we select urban planning maps as a representative sample and investigate the feasibility of utilizing U-shape fully convolutional based architecture to perform end-to-end map semantic segmentation. The experimental results obtained from the test area in Shibuya district, Tokyo, demonstrate that our proposed method could achieve …
Technical University of Munich（慕尼黑工业大学）；Klinikum rechts der Isar（慕尼黑 Rechts der Isar 医院）；University of Zurich（苏黎世大学）
Automatic localization and segmentation of organs-at-risk (OAR) in CT are essential pre-processing steps in medical image analysis tasks, such as radiation therapy planning. For instance, the segmentation of OAR surrounding tumors enables the maximization of radiation to the tumor area without compromising the healthy tissues. However, the current medical workflow requires manual delineation of OAR, which is prone to errors and is annotator-dependent. In this work, we aim to introduce a unified 3D pipeline for OAR localization-segmentation rather than novel localization or segmentation archite…
Spine Vision X-Ray Image based GUI Planning of Pedicle Screws Using Enhanced YOLOv5 for Vertebrae Segmentation
Healthcare Technology Innovation Centre (HTIC), Indian Institute of Technology Madras（印度理工学院马德拉斯分校医疗技术创新中心）；Indian Institute of Technology Madras（印度理工学院马德拉斯分校）；Department of Electrical Engineering, Indian Institute of Technology Madras（印度理工学院马德拉斯分校电气工程系）
Efficient data collection methods play a major role in helping us better understand the Earth and its ecosystems. In many applications, the usage of unmanned aerial vehicles (UAVs) for monitoring and remote sensing is rapidly gaining momentum due to their high mobility, low cost, and flexible deployment. A key challenge is planning missions to maximize the value of acquired data in large environments given flight time limitations. This is, for example, relevant for monitoring agricultural fields. This paper addresses the problem of adaptive path planning for accurate semantic segmentation of u…
This paper proposes an extension to the Dual Branch Prior-Net for sparse view interventional CBCT reconstruction incorporating a high quality planning scan. An additional head learns to segment interventional instruments and thus guides the reconstruction task. The prior scans are misaligned by up to +-5deg in-plane during training. Experiments show that the proposed model, Dual Branch Prior-SegNet, significantly outperforms any other evaluated model by >2.8dB PSNR. It also stays robust wrt. rotations of up to +-5.5deg.
Semantic Segmentation for Preoperative Planning in Transcatheter Aortic Valve Replacement
Karlsruhe Institute of Technology, Karlsruhe, Germany（卡尔斯鲁厄理工学院）；Helmholtz Information and Data Science School for Health, Germany（海德堡信息与数据科学健康学校）；Department of Thoracic and Cardiovascular Surgery, Heart Centre Lahr, Germany（胸外科与心血管外科部门，劳赫心脏中心）
Memorial Sloan-Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）
Although deep convolutional networks have been widely studied for head and neck (HN) organs at risk (OAR) segmentation, their use for routine clinical treatment planning is limited by a lack of robustness to imaging artifacts, low soft tissue contrast on CT, and the presence of abnormal anatomy. In order to address these challenges, we developed a computationally efficient nested block self-attention (NBSA) method that can be combined with any convolutional network. Our method achieves computational efficiency by performing non-local calculations within memory blocks of fixed spatial extent. C…
Friedrich-Alexander-Universität Erlangen-Nürnberg（弗里德里希-亚历山大-埃尔兰根-纽伦堡大学）；Siemens Healthcare GmbH（西门子医疗保健有限公司）；Hochschule Furtwangen（富特旺根应用技术大学）；BG Trauma Center Ludwigshafen（路德维希港BG创伤中心）
X-ray based measurement and guidance are commonly used tools in orthopaedic surgery to facilitate a minimally invasive workflow. Typically, a surgical planning is first performed using knowledge of bone morphology and anatomical landmarks. Information about bone location then serves as a prior for registration during overlay of the planning on intra-operative X-ray images. Performing these steps manually however is prone to intra-rater/inter-rater variability and increases task complexity for the surgeon. To remedy these issues, we propose an automatic framework for planning and subsequent ove…
Leveraging Semantic Asymmetry for Precise Gross Tumor Volume Segmentation of Nasopharyngeal Carcinoma in Planning CT
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；The University of Hong Kong（香港大学）；The First Affiliated Hospital, Zhejiang University（浙江大学第一附属医院）；Hupan Lab（华盼实验室）
Tumor treating fields (TTFields) is an FDA approved therapy for the treatment of Gliobastoma Multiform (GBM) and currently being investigated for additional tumor types. TTFields are delivered to the tumor through the placement of transducer arrays (TAs) placed on the patient scalp. The positions of the TAs are associated with treatment outcomes via simulations of the electric fields. Therefore, we are currently developing a method for recommending optimal placement of TAs. A key step to achieve this goal is to correctly segment the head into tissues of similar electrical properties. Visual in…
Segmentation of mandibles in CT scans during virtual surgical planning is crucial for 3D surgical planning in order to obtain a detailed surface representation of the patients bone. Automatic segmentation of mandibles in CT scans is a challenging task due to large variation in their shape and size between individuals. In order to address this challenge we propose a convolutional neural network approach for mandible segmentation in CT scans by considering the continuum of anatomical structures through different planes. The proposed convolutional neural network adopts the architecture of the U-N…
Northwestern University（西北大学）；University of Chicago（芝加哥大学）
Head and Neck (H\&N) organ-at-risk (OAR) and tumor segmentations are essential components of radiation therapy planning. The varying anatomic locations and dimensions of H\&N nodal Gross Tumor Volumes (GTVn) and H\&N primary gross tumor volume (GTVp) are difficult to obtain due to lack of accurate and reliable delineation methods. The downstream effect of incorrect segmentation can result in unnecessary irradiation of normal organs. Towards a fully automated radiation therapy planning algorithm, we explore the efficacy of multi-scale fusion based deep learning architectures for accurately segm…
Université Côte d’Azur（蔚蓝海岸大学）；Inria（法国国家信息与自动化研究所）；Centre Antoine Lacassagne（安托万·拉卡桑中心）
Planning of radiotherapy involves accurate segmentation of a large number of organs at risk, i.e. organs for which irradiation doses should be minimized to avoid important side effects of the therapy. We propose a deep learning method for segmentation of organs at risk inside the brain region, from Magnetic Resonance (MR) images. Our system performs segmentation of eight structures: eye, lens, optic nerve, optic chiasm, pituitary gland, hippocampus, brainstem and brain. We propose an efficient algorithm to train neural networks for an end-to-end segmentation of multiple and non-exclusive class…
Georgia Institute of Technology（佐治亚理工学院）；Memorial Sloan-Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）；Peking University Cancer Hospital（北京大学肿瘤医院）
In current clinical practice, noisy and artifact-ridden weekly cone-beam computed tomography (CBCT) images are only used for patient setup during radiotherapy. Treatment planning is done once at the beginning of the treatment using high-quality planning CT (pCT) images and manual contours for organs-at-risk (OARs) structures. If the quality of the weekly CBCT images can be improved while simultaneously segmenting OAR structures, this can provide critical information for adapting radiotherapy mid-treatment as well as for deriving biomarkers for treatment response. Using a novel physics-based da…
University College London（伦敦大学学院）；King’s College London（伦敦国王学院）；National University of Singapore（新加坡国立大学）
Multi-task neural network architectures provide a mechanism that jointly integrates information from distinct sources. It is ideal in the context of MR-only radiotherapy planning as it can jointly regress a synthetic CT (synCT) scan and segment organs-at-risk (OAR) from MRI. We propose a probabilistic multi-task network that estimates: 1) intrinsic uncertainty through a heteroscedastic noise model for spatially-adaptive task loss weighting and 2) parameter uncertainty through approximate Bayesian inference. This allows sampling of multiple segmentations and synCTs that share their network repr…
Mannheim University of Applied Science（曼海姆应用科技大学）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）；DZHK (German Centre for Cardiovascular Research)（德国心血管研究中心）；German Heart Centre Munich（慕尼黑德国心脏中心）；Hannover Medical School（汉诺威医学院）；Heidelberg University Hospital（海德堡大学医院）
Planning the optimal time of intervention for pulmonary valve replacement surgery in patients with the congenital heart disease Tetralogy of Fallot (TOF) is mainly based on ventricular volume and function according to current guidelines. Both of these two biomarkers are most reliably assessed by segmentation of 3D cardiac magnetic resonance (CMR) images. In several grand challenges in the last years, U-Net architectures have shown impressive results on the provided data. However, in clinical practice, data sets are more diverse considering individual pathologies and image properties derived fr…
Deep-learning Segmentation of Small Volumes in CT images for Radiotherapy Treatment Planning
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；OSF HealthCare（OSF医疗保健）；IFC-National Research Council（IFC国家研究委员会）；Terarecon Inc.（Terarecon公司）
Department of Biomedical Engineering, University of Basel（巴塞尔大学生物医学工程系）；Department of Biomedicine, Musculoskeletal Research, University of Basel（巴塞尔大学生物医学系肌肉骨骼研究部）；University of Basel（巴塞尔大学）
Segmentation of distinct bones plays a crucial role in diagnosis, planning, navigation, and the assessment of bone metastasis. It supplies semantic knowledge to visualisation tools for the planning of surgical interventions and the education of health professionals. Fully supervised segmentation of 3D data using Deep Learning methods has been extensively studied for many tasks but is usually restricted to distinguishing only a handful of classes. With 125 distinct bones, our case includes many more labels than typical 3D segmentation tasks. For this reason, the direct adaptation of most establ…
Transformer-based cardiac substructure segmentation from contrast and non-contrast computed tomography for radiotherapy planning
Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）；Medical Center-University of Freiburg, Faculty of Medicine, University of Freiburg（弗莱堡大学医学院大学医学中心）；German Cancer Consortium (DKTK)（德国癌症联盟（DKTK））
University of Tehran（德黑兰大学）；Nara Institute of Science and Technology（奈良科学技术研究所）；Osaka University Graduate School of Medicine（大阪大学医学院）
With the increasing usage of radiograph images as a most common medical imaging system for diagnosis, treatment planning, and clinical studies, it is increasingly becoming a vital factor to use machine learning-based systems to provide reliable information for surgical pre-planning. Segmentation of pelvic bone in radiograph images is a critical preprocessing step for some applications such as automatic pose estimation and disease detection. However, the encoder-decoder style network known as U-Net has demonstrated limited results due to the challenging complexity of the pelvic shapes, especial…
SegRap2025: A Benchmark of Gross Tumor Volume and Lymph Node Clinical Target Volume Segmentation for Radiotherapy Planning of Nasopharyngeal Carcinoma
School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China（电子科技大学机械与电子工程学院）；Department of Radiation Oncology, Sichuan Cancer Center, Radiation Oncology Key Laboratory of Sichuan Province, Sichuan Clinical Research Center for Cancer, Sichuan Cancer Hospital and Institute, University of Electronic Science and Technology of China（四川省肿瘤医院放射肿瘤科）；Shanghai AI Lab, Shanghai, China（上海人工智能实验室）；The Affiliated Hospital of Qingdao University, Qingdao, China（青岛大学附属医院）；Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China（香港中文大学计算机科学与工程系）；Faculty of Information Technology, Monash University, Melbourne, Australia（墨尔本大学信息科技学院）；Department of Computing and Department of Brain Sciences, Imperial College London, United Kingdom（伦敦帝国理工学院计算机系和脑科学系）；School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an, China（西北工业大学计算机科学与工程学院）；UMIT Tirol - Private University for Health Sciences and Health Technology, Austria（蒂罗尔大学（私人健康科学与健康技术大学））；School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）；National Heart and Lung Institute, Imperial College London, London, United Kingdom（伦敦帝国理工学院国家心脏和肺研究所）；Hawkes Institute, Department of Computer Science, University College London, London, United Kingdom（伦敦大学学院霍克斯研究所）；Shanghai MediWorks Precision Instruments Co., Ltd., China（上海 MediWorks 精密仪器有限公司）；School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China（上海交通大学自动化与智能感知学院）；Department of Systems Hub, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China（香港科技大学（广州）系统中心）
Fraunhofer MEVIS（弗劳恩霍夫医学图像计算研究所）；University of Bremen（不来梅大学）；Radboud University（拉德堡德大学）；Yokohama City University（横滨市立大学）
Various approaches for liver segmentation in CT have been proposed: Besides statistical shape models, which played a major role in this research area, novel approaches on the basis of convolutional neural networks have been introduced recently. Using a set of 219 liver CT datasets with reference segmentations from liver surgery planning, we evaluate the performance of several neural network classifiers based on 2D and 3D U-net architectures. An interesting observation is that slice-wise approaches perform surprisingly well, with mean and median Dice coefficients above 0.97, and may be preferab…
TU Darmstadt（达姆施塔特工业大学）；Istanbul Technical University（伊斯坦布尔理工大学）；King's College（国王学院）
Since the advent of U-Net, fully convolutional deep neural networks and its many variants have completely changed the modern landscape of deep learning based medical image segmentation. However, the over dependence of these methods on pixel level classification and regression has been identified early on as a problem. Especially when trained on medical databases with sparse available annotation, these methods are prone to generate segmentation artifacts such as fragmented structures, topological inconsistencies and islands of pixel. These artefacts are especially problematic in medical imaging…
Automatic segmentation of vestibular schwannoma (VS) and cochlea from magnetic resonance imaging can facilitate VS treatment planning. Unsupervised segmentation methods have shown promising results without requiring the time-consuming and laborious manual labeling process. In this paper, we present an approach for VS and cochlea segmentation in an unsupervised domain adaptation setting. Specifically, we first develop a cross-site cross-modality unpaired image translation strategy to enrich the diversity of the synthesized data. Then, we devise a rule-based offline augmentation technique to fur…
Yokohama City University Medical Center（横滨市立大学医学中心）
We present a fully automatic method employing convolutional neural networks based on the 2D U-net architecture and random forest classifier to solve the automatic liver lesion segmentation problem of the ISBI 2017 Liver Tumor Segmentation Challenge (LiTS). In order to constrain the ROI in which the tumors could be located, a liver segmentation is performed first. For the organ segmentation, an ensemble of convolutional networks is trained to segment a liver using a set of 179 liver CT datasets from liver surgery planning. Inside of the liver ROI a neural network, trained using 127 challenge tr…
SegRap2023: A Benchmark of Organs-at-Risk and Gross Tumor Volume Segmentation for Radiotherapy Planning of Nasopharyngeal Carcinoma
University of Electronic Science and Technology of China（电子科技大学）；Sichuan Cancer Hospital & Institute（四川省肿瘤医院（研究所））；Shanghai AI Lab（上海人工智能实验室）；Canon Medical Systems (China) Co. Ltd.（佳能医疗系统（中国）有限公司）；Stockholm University（斯德哥尔摩大学）；KTH（瑞典皇家理工学院）；Northwestern Polytechnical University（西北工业大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Liver lesion segmentation is an important step for liver cancer diagnosis, treatment planning and treatment evaluation. LiTS (Liver Tumor Segmentation Challenge) provides a common testbed for comparing different automatic liver lesion segmentation methods. We participate in this challenge by developing a deep convolutional neural network (DCNN) method. The particular DCNN model works in 2.5D in that it takes a stack of adjacent slices as input and produces the segmentation map corresponding to the center slice. The model has 32 layers in total and makes use of both long range concatenation con…
Veeva Systems（Veeva系统公司）；Lyft Inc（来福车公司）；Massachusetts Institute of Technology（麻省理工学院）
Semantic segmentation is in-demand in satellite imagery processing. Because of the complex environment, automatic categorization and segmentation of land cover is a challenging problem. Solving it can help to overcome many obstacles in urban planning, environmental engineering or natural landscape monitoring. In this paper, we propose an approach for automatic multi-class land segmentation based on a fully convolutional neural network of feature pyramid network (FPN) family. This network is consisted of pre-trained on ImageNet Resnet50 encoder and neatly developed decoder. Based on validation …
Multimodal brain tumor segmentation challenge (BraTS) brings together researchers to improve automated methods for 3D MRI brain tumor segmentation. Tumor segmentation is one of the fundamental vision tasks necessary for diagnosis and treatment planning of the disease. Previous years winning methods were all deep-learning based, thanks to the advent of modern GPUs, which allow fast optimization of deep convolutional neural network architectures. In this work, we explore best practices of 3D semantic segmentation, including conventional encoder-decoder architecture, as well combined loss functio…
King’s College London（伦敦国王学院）；University College London（伦敦大学学院）
Automatic brain tumor segmentation plays an important role for diagnosis, surgical planning and treatment assessment of brain tumors. Deep convolutional neural networks (CNNs) have been widely used for this task. Due to the relatively small data set for training, data augmentation at training time has been commonly used for better performance of CNNs. Recent works also demonstrated the usefulness of using augmentation at test time, in addition to training time, for achieving more robust predictions. We investigate how test-time augmentation can improve CNNs' performance for brain tumor segment…
Data Science Institute, Imperial College London（伦敦帝国理工学院数据科学研究所）；National Heart and Lung Institute, Imperial College London（伦敦帝国理工学院国家心肺研究所）
A major challenge in brain tumor treatment planning and quantitative evaluation is determination of the tumor extent. The noninvasive magnetic resonance imaging (MRI) technique has emerged as a front-line diagnostic tool for brain tumors without ionizing radiation. Manual segmentation of brain tumor extent from 3D MRI volumes is a very time-consuming task and the performance is highly relied on operator's experience. In this context, a reliable fully automatic segmentation method for the brain tumor segmentation is necessary for an efficient measurement of the tumor extent. In this study, we p…
Worcester Polytechnic Institute（伍斯特理工学院）；Department of Electrical and Computer Engineering（电气与计算机工程系）
This paper presents an accurate and fast algorithm for road segmentation using convolutional neural network (CNN) and gated recurrent units (GRU). For autonomous vehicles, road segmentation is a fundamental task that can provide the drivable area for path planning. The existing deep neural network based segmentation algorithms usually take a very deep encoder-decoder structure to fuse pixels, which requires heavy computations, large memory and long processing time. Hereby, a CNN-GRU network model is proposed and trained to perform road segmentation using data captured by the front camera of a …
