Unsupervised domain adaptation (UDA) between two significantly disparate domains to learn high-level semantic alignment is a crucial yet challenging task.~To this end, in this work, we propose exploiting low-level edge information to facilitate the adaptation as a precursor task, which has a small cross-domain gap, compared with semantic segmentation.~The precise contour then provides spatial information to guide the semantic adaptation. More specifically, we propose a multi-task framework to learn a contouring adaptation network along with a semantic segmentation adaptation network, which takes both magnetic resonance imaging (MRI) slice and its initial edge map as input.~These two networks are jointly trained with source domain labels, and the feature and edge map level adversarial learning is carried out for cross-domain alignment. In addition, self-entropy minimization is incorporated to further enhance segmentation performance. We evaluated our framework on the BraTS2018 database for cross-modality segmentation of brain tumors, showing the validity and superiority of our approach, compared with competing methods.
Three-dimensional (3D) integrated renal structures (IRS) segmentation is important in clinical practice. With the advancement of deep learning techniques, many powerful frameworks focusing on medical image segmentation are proposed. In this challenge, we utilized the nnU-Net framework, which is the state-of-the-art method for medical image segmentation. To reduce the outlier prediction for the tumor label, we combine contour regularization (CR) loss of the tumor label with Dice loss and cross-entropy loss to improve this phenomenon.
The Chinese University of Hong Kong（香港中文大学）；Imperial College Longdon（伦敦帝国理工学院）；Imsight Medical Technology Co. Ltd.（艾迈医疗科技有限公司）；The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）
Accurate medical image segmentation commonly requires effective learning of the complementary information from multimodal data. However, in clinical practice, we often encounter the problem of missing imaging modalities. We tackle this challenge and propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities. Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code, which uniquely sticks to each modality, and the modality-invariant content code, which absorbs multimodal information for the s…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Provincial People’s Hospital（广东省人民医院）；Guangdong Academy of Medical Sciences（广东省医学科学院）
Brain tumor segmentation based on multi-modal magnetic resonance imaging (MRI) plays a pivotal role in assisting brain cancer diagnosis, treatment, and postoperative evaluations. Despite the achieved inspiring performance by existing automatic segmentation methods, multi-modal MRI data are still unavailable in real-world clinical applications due to quite a few uncontrollable factors (e.g. different imaging protocols, data corruption, and patient condition limitations), which lead to a large performance drop during practical applications. In this work, we propose a Deeply supervIsed knowledGE …
Deep Learning is a promising approach to either automate or simplify several tasks in the healthcare domain. In this work, we introduce SegAN-CAT, an approach to brain tumor segmentation in Magnetic Resonance Images (MRI), based on Adversarial Networks. In particular, we extend SegAN, successfully applied to the same task in a previous work, in two respects: (i) we used a different model input and (ii) we employed a modified loss function to train the model. We tested our approach on two large datasets, made available by the Brain Tumor Image Segmentation Benchmark (BraTS). First, we trained a…
With the development of image segmentation in computer vision, biomedical image segmentation have achieved remarkable progress on brain tumor segmentation and Organ At Risk (OAR) segmentation. However, most of the research only uses single modality such as Computed Tomography (CT) scans while in real world scenario doctors often use multiple modalities to get more accurate result. To better leverage different modalities, we have collected a large dataset consists of 136 cases with CT and MR images which diagnosed with nasopharyngeal cancer. In this paper, we propose to use Generative Adversari…
University of Leicester（莱斯特大学）；University of Lancaster（兰卡斯特大学）；Queen Mary, University of London（伦敦大学玛丽女王学院）；Xidian University（西安电子科技大学）；Bielefeld University（比勒费尔德大学）
Brain tumor segmentation is one of the most challenging problems in medical image analysis. The goal of brain tumor segmentation is to generate accurate delineation of brain tumor regions. In recent years, deep learning methods have shown promising performance in solving various computer vision problems, such as image classification, object detection and semantic segmentation. A number of deep learning based methods have been applied to brain tumor segmentation and achieved promising results. Considering the remarkable breakthroughs made by state-of-the-art technologies, we use this survey to …
Benefiting from color independence, illumination invariance and location discrimination attributed by the depth map, it can provide important supplemental information for extracting salient objects in complex environments. However, high-quality depth sensors are expensive and can not be widely applied. While general depth sensors produce the noisy and sparse depth information, which brings the depth-based networks with irreversible interference. In this paper, we propose a novel multi-task and multi-modal filtered transformer (MMFT) network for RGB-D salient object detection (SOD). Specificall…
Tunghai University（东海大学）；Changhua Christian Hospital（彰化基督教医院）
Malignant and benign breast tumors present differently in their shape and size on sonography. Morphological information provided by tumor contours are important in clinical diagnosis. However, ultrasound images contain noises and tissue texture; clinical diagnosis thus highly depends on the experience of physicians. The manual way to sketch three-dimensional (3D) contours of breast tumor is a time-consuming and complicate task. If automatic contouring could provide a precise breast tumor contour that might assist physicians in making an accurate diagnosis. This study presents an efficient meth…
Multi-Modal Brain Tumor Segmentation via 3D Multi-Scale Self-attention and Cross-attention
University of Electronic Science and Technology of China（电子科技大学）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；School of Information and Software Engineering, University of Electronic Science and Technology of China（电子科技大学信息与软件工程学院）
University of Waterloo（滑铁卢大学）；Sunnybrook Health Sciences Centre（森尼布鲁克健康科学中心）
Marking tumors and organs is a challenging task suffering from both inter- and intra-observer variability. The literature quantifies observer variability by generating consensus among multiple experts when they mark the same image. Automatically building consensus contours to establish quality assurance for image segmentation is presently absent in the clinical practice. As the \emph{big data} becomes more and more available, techniques to access a large number of existing segments of multiple experts becomes possible. Fast algorithms are, hence, required to facilitate the search for similar c…
Nanjing University of Science and Technology（南京理工大学）；Nanjing University（南京大学）
Automatic segmentation of head and neck tumors plays an important role in radiomics analysis. In this short paper, we propose an automatic segmentation method for head and neck tumors from PET and CT images based on the combination of convolutional neural networks (CNNs) and hybrid active contours. Specifically, we first introduce a multi-channel 3D U-Net to segment the tumor with the concatenated PET and CT images. Then, we estimate the segmentation uncertainty by model ensembles and define a segmentation quality score to select the cases with high uncertainties. Finally, we develop a hybrid …
Heudiasyc（Heudiasyc实验室）；CNRS（法国国家科学研究中心）；Université de technologie de Compiègne（贡比涅技术大学）；Quantif（Quantif团队）；LITIS（LITIS实验室）；University of Rouen Normandy（鲁昂诺曼底大学）；Institut universitaire de France（法国大学研究院）；Henri Becquerel Cancer Center（亨利·贝克勒尔癌症中心）
As information sources are usually imperfect, it is necessary to take into account their reliability in multi-source information fusion tasks. In this paper, we propose a new deep framework allowing us to merge multi-MR image segmentation results using the formalism of Dempster-Shafer theory while taking into account the reliability of different modalities relative to different classes. The framework is composed of an encoder-decoder feature extraction module, an evidential segmentation module that computes a belief function at each voxel for each modality, and a multi-modality evidence fusion…
multiPI-TransBTS: A Multi-Path Learning Framework for Brain Tumor Image Segmentation Based on Multi-Physical Information
Chongqing University of Posts and Telecommunications（重庆邮电大学）；Chongqing Engineering Research Center of Software Quality Assurance, Testing and Assessment（重庆市软件质量保证与测评工程技术研究中心）
Shandong Normal University（山东师范大学）；Western University（韦仕敦大学）
Simultaneous segmentation and detection of liver tumors (hemangioma and hepatocellular carcinoma (HCC)) by using multi-modality non-contrast magnetic resonance imaging (NCMRI) are crucial for the clinical diagnosis. However, it is still a challenging task due to: (1) the HCC information on NCMRI is invisible or insufficient makes extraction of liver tumors feature difficult; (2) diverse imaging characteristics in multi-modality NCMRI causes feature fusion and selection difficult; (3) no specific information between hemangioma and HCC on NCMRI cause liver tumors detection difficult. In this stu…
One of the main requirements of tumor extraction is the annotation and segmentation of tumor boundaries correctly. For this purpose, we present a threefold deep learning architecture. First classifiers are implemented with a deep convolutional neural network(CNN) andsecond a region-based convolutional neural network (R-CNN) is performed on the classified images to localize the tumor regions of interest. As the third and final stage, the concentratedtumor boundary is contoured for the segmentation process by using the Chan-Vesesegmentation algorithm. As the typical edge detection algorithms bas…
With the advent of advancements in deep learning approaches, such as deep convolution neural network, residual neural network, adversarial network; U-Net architectures are most widely utilized in biomedical image segmentation to address the automation in identification and detection of the target regions or sub-regions. In recent studies, U-Net based approaches have illustrated state-of-the-art performance in different applications for the development of computer-aided diagnosis systems for early diagnosis and treatment of diseases such as brain tumor, lung cancer, alzheimer, breast cancer, et…
In this paper we present a new method for motion tracking of tumors in liver ultrasound image sequences. Our algorithm has two main steps. In the first step, we apply mean shift algorithm with multiple features to estimate the center of the target in each frame. Target in the first frame is defined using an ellipse. Edge, texture, and intensity features are extracted from the first frame, and then mean shift algorithm is applied to each feature separately to find the center of ellipse related to that feature in the next frame. The center of ellipse will be the weighted average of these centers…
School of Mechano-Electronic Engineering, Xidian University（西安电子科技大学机电工程学院）；Computer Science Department, Aberystwyth University（阿伯里斯特威斯大学计算机科学系）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；Deepwise AI Lab（深睿人工智能实验室）
Recent advances in machine learning and prevalence of digital medical images have opened up an opportunity to address the challenging brain tumor segmentation (BTS) task by using deep convolutional neural networks. However, different from the RGB image data that are very widespread, the medical image data used in brain tumor segmentation are relatively scarce in terms of the data scale but contain the richer information in terms of the modality property. To this end, this paper proposes a novel cross-modality deep feature learning framework to segment brain tumors from the multi-modality MRI d…
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Multi-modal magnetic resonance (MR) imaging provides great potential for diagnosing and analyzing brain gliomas. In clinical scenarios, common MR sequences such as T1, T2 and FLAIR can be obtained simultaneously in a single scanning process. However, acquiring contrast enhanced modalities such as T1ce requires additional time, cost, and injection of contrast agent. As such, it is clinically meaningful to develop a method to synthesize unavailable modalities which can also be used as additional inputs to downstream tasks (e.g., brain tumor segmentation) for performance enhancing. In this work, …
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Lenovo Research（联想研究院）
Automatic brain tumor segmentation from multi-modality Magnetic Resonance Images (MRI) using deep learning methods plays an important role in assisting the diagnosis and treatment of brain tumor. However, previous methods mostly ignore the latent relationship among different modalities. In this work, we propose a novel end-to-end Modality-Pairing learning method for brain tumor segmentation. Paralleled branches are designed to exploit different modality features and a series of layer connections are utilized to capture complex relationships and abundant information among modalities. We also us…
A Unified Attention U-Net Framework for Cross-Modality Tumor Segmentation in MRI and CT
Department of Computer Science, New Mexico State University（计算机科学系，新墨西哥州立大学）；Department of Plant and Environmental Sciences, New Mexico State University（植物与环境科学系，新墨西哥州立大学）
Magnetic Resonance (MR) images of different modalities can provide complementary information for clinical diagnosis, but whole modalities are often costly to access. Most existing methods only focus on synthesizing missing images between two modalities, which limits their robustness and efficiency when multiple modalities are missing. To address this problem, we propose a multi-modality generative adversarial network (MGAN) to synthesize three high-quality MR modalities (FLAIR, T1 and T1ce) from one MR modality T2 simultaneously. The experimental results show that the quality of the synthesize…
VRVis Zentrum für Virtual Reality und Visualisierung Forschungs-GmbH（VRVis虚拟现实与可视化研究有限公司）；Center for Medical Physics and Biomedical Engineering, Medical University of Vienna（维也纳医科大学医学物理与生物医学工程中心）；Center for Nuclear Medicine, Clinical Center of Serbia（塞尔维亚临床中心核医学中心）
Tumor segmentation in multimodal medical images has seen a growing trend towards deep learning based methods. Typically, studies dealing with this topic fuse multimodal image data to improve the tumor segmentation contour for a single imaging modality. However, they do not take into account that tumor characteristics are emphasized differently by each modality, which affects the tumor delineation. Thus, the tumor segmentation is modality- and task-dependent. This is especially the case for soft tissue sarcomas, where, due to necrotic tumor tissue, the segmentation differs vastly. Closing this …
AnatomicalNets: A Multi-Structure Segmentation and Contour-Based Distance Estimation Pipeline for Clinically Grounded Lung Cancer T-Staging
Department of Robotics and Mechatronics Engineering, University of Dhaka（达卡大学机器人与机电工程系）；Department of Electrical and Electronics Engineering, University of Dhaka（达卡大学电气与电子工程系）；Department of Electrical Engineering, Qatar University（卡塔尔大学电气工程系）；Department of Radiology, Hamad Medical Corporation（哈马德医疗集团放射科）；Department of Biomedical Technology, Prince Sattam Bin Abdulaziz University（普林斯·萨塔姆·本·阿卜杜勒阿齐兹大学生物医学技术系）
CCSD: Cross-Modal Compositional Self-Distillation for Robust Brain Tumor Segmentation with Missing Modalities
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（上海智能自主系统研究院，同济大学）；College of Biomedical Engineering, Fudan University（生物医学工程学院，复旦大学）；School of Medicine, Tongji University（医学院，同济大学）；College of Information Engineering, China Jiliang University（信息工程学院，中国嘉兴大学）
South China University of Technology（华南理工大学）；Rochester Institute of Technology（罗切斯特理工学院）
Due to the difficulties of obtaining multimodal paired images in clinical practice, recent studies propose to train brain tumor segmentation models with unpaired images and capture complementary information through modality translation. However, these models cannot fully exploit the complementary information from different modalities. In this work, we thus present a novel two-step (intra-modality and inter-modality) curriculum disentanglement learning framework to effectively utilize privileged semi-paired images, i.e. limited paired images that are only available in training, for brain tumor …
Technical University of Denmark（丹麦技术大学）；Skåne University Hospital（斯科讷大学医院）；Danish Research Centre for Magnetic Resonance（丹麦磁共振研究中心）；Copenhagen University Hospital Hvidovre（哥本哈根大学医院维兹奥勒医院）；Copenhagen University Hospital Rigshospitalet（哥本哈根大学医院国立医院）；University College London（伦敦大学学院）；UCLH NHS Foundation Trust（伦敦大学学院医院NHS基金会信托）；National Hospital for Neurology and Neurosurgery（国立神经病学与神经外科医院）
In this paper we present a method for simultaneously segmenting brain tumors and an extensive set of organs-at-risk for radiation therapy planning of glioblastomas. The method combines a contrast-adaptive generative model for whole-brain segmentation with a new spatial regularization model of tumor shape using convolutional restricted Boltzmann machines. We demonstrate experimentally that the method is able to adapt to image acquisitions that differ substantially from any available training data, ensuring its applicability across treatment sites; that its tumor segmentation accuracy is compara…
Tianjin University（天津大学）；The University of Hong Kong（香港大学）；Huanhu Hospital（环湖医院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）
Multi-modal MR imaging is routinely used in clinical practice to diagnose and investigate brain tumors by providing rich complementary information. Previous multi-modal MRI segmentation methods usually perform modal fusion by concatenating multi-modal MRIs at an early/middle stage of the network, which hardly explores non-linear dependencies between modalities. In this work, we propose a novel Nested Modality-Aware Transformer (NestedFormer) to explicitly explore the intra-modality and inter-modality relationships of multi-modal MRIs for brain tumor segmentation. Built on the transformer-based…
Mind the Gap: Promoting Missing Modality Brain Tumor Segmentation with Alignment
School of Robotics Xi’an Jiaotong-Liverpool University Suzhou, China；School of Advanced Technology Xi'an Jiaotong-Liverpool University Suzhou, China；School of Advanced Technology Xi’an Jiaotong-Liverpool University Suzhou, China；Data Science Research Center Duke Kunshan University Suzhou, China
INSA de Rouen（鲁昂国立应用科学学院）；Université de Rouen Normandie（鲁昂诺曼底大学）；Normandie Univ（诺曼底大学）；Henri Becquerel Cancer Center（亨利·贝克勒尔癌症中心）
Using multimodal Magnetic Resonance Imaging (MRI) is necessary for accurate brain tumor segmentation. The main problem is that not all types of MRIs are always available in clinical exams. Based on the fact that there is a strong correlation between MR modalities of the same patient, in this work, we propose a novel brain tumor segmentation network in the case of missing one or more modalities. The proposed network consists of three sub-networks: a feature-enhanced generator, a correlation constraint block and a segmentation network. The feature-enhanced generator utilizes the available modali…
South China University of Technology（华南理工大学）；Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences（广东省人民医院，广东省医学科学院）
Brain tumor segmentation (BTS) in magnetic resonance image (MRI) is crucial for brain tumor diagnosis, cancer management and research purposes. With the great success of the ten-year BraTS challenges as well as the advances of CNN and Transformer algorithms, a lot of outstanding BTS models have been proposed to tackle the difficulties of BTS in different technical aspects. However, existing studies hardly consider how to fuse the multi-modality images in a reasonable manner. In this paper, we leverage the clinical knowledge of how radiologists diagnose brain tumors from multiple MRI modalities…
ARC Training Centre for Innovative BioEngineering（ARC创新生物工程培训中心）
Distant metastases (DM) refer to the dissemination of tumors, usually, beyond the organ where the tumor originated. They are the leading cause of death in patients with soft-tissue sarcomas (STSs). Positron emission tomography-computed tomography (PET-CT) is regarded as the imaging modality of choice for the management of STSs. It is difficult to determine from imaging studies which STS patients will develop metastases. 'Radiomics' refers to the extraction and analysis of quantitative features from medical images and it has been employed to help identify such tumors. The state-of-the-art in ra…
University at Buffalo, The State University of New York（纽约州立大学布法罗分校）
Brain Tumor Segmentation from magnetic resonance imaging (MRI) is a critical technique for early diagnosis. However, rather than having complete four modalities as in BraTS dataset, it is common to have missing modalities in clinical scenarios. We design a brain tumor segmentation algorithm that is robust to the absence of any modality. Our network includes a channel-independent encoding path and a feature-fusion decoding path. We use self-supervised training through channel dropout and also propose a novel domain adaptation method on feature maps to recover the information from the missing ch…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Lenovo Research（联想研究院）；Lenovo Ltd.（联想集团有限公司）
Liver cancer is one of the most common cancers worldwide. Due to inconspicuous texture changes of liver tumor, contrast-enhanced computed tomography (CT) imaging is effective for the diagnosis of liver cancer. In this paper, we focus on improving automated liver tumor segmentation by integrating multi-modal CT images. To this end, we propose a novel mutual learning (ML) strategy for effective and robust multi-modal liver tumor segmentation. Different from existing multi-modal methods that fuse information from different modalities by a single model, with ML, an ensemble of modality-specific mo…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent（腾讯）；University of California, Los Angeles（加州大学洛杉矶分校）；Lenovo Research（联想研究院）
Accurate brain tumor segmentation from Magnetic Resonance Imaging (MRI) is desirable to joint learning of multimodal images. However, in clinical practice, it is not always possible to acquire a complete set of MRIs, and the problem of missing modalities causes severe performance degradation in existing multimodal segmentation methods. In this work, we present the first attempt to exploit the Transformer for multimodal brain tumor segmentation that is robust to any combinatorial subset of available modalities. Concretely, we propose a novel multimodal Medical Transformer (mmFormer) for incompl…
Memorial Sloan-Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）
Lack of large expert annotated MR datasets makes training deep learning models difficult. Therefore, a cross-modality (MR-CT) deep learning segmentation approach that augments training data using pseudo MR images produced by transforming expert-segmented CT images was developed. Eighty-One T2-weighted MRI scans from 28 patients with non-small cell lung cancers were analyzed. Cross-modality prior encoding the transformation of CT to pseudo MR images resembling T2w MRI was learned as a generative adversarial deep learning model. This model augmented training data arising from 6 expert-segmented …
