Merging-Diverging Hybrid Transformer Networks for Survival Prediction in Head and Neck Cancer
School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）；Royal Prince Alfred Hospital（皇家阿尔弗雷德王子医院）；Med-X Research Institute, Shanghai Jiao Tong University（上海交通大学Med-X研究院）
Mingyuan Meng, Lei Bi, Michael Fulham, Dagan Feng, Jinman Kim
备注：
Early Accepted at International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2023)
发表信息：
International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), pp. 400-410, 2023
Survival prediction is crucial for cancer patients as it provides early prognostic information for treatment planning. Recently, deep survival models based on deep learning and medical images have shown promising performance for survival prediction. However, existing deep survival models are not well developed in utilizing multi-modality images (e.g., PET-CT) and in extracting region-specific information (e.g., the prognostic information in Primary Tumor (PT) and Metastatic Lymph Node (MLN) regions). In view of this, we propose a merging-diverging learning framework for survival prediction from multi-modality images. This framework has a merging encoder to fuse multi-modality information and a diverging decoder to extract region-specific information. In the merging encoder, we propose a Hybrid Parallel Cross-Attention (HPCA) block to effectively fuse multi-modality features via parallel convolutional layers and cross-attention transformers. In the diverging decoder, we propose a Region-specific Attention Gate (RAG) block to screen out the features related to lesion regions. Our framework is demonstrated on survival prediction from PET-CT images in Head and Neck (H&N) cancer, by designing an X-shape merging-diverging hybrid transformer network (named XSurv). Our XSurv combines the complementary information in PET and CT images and extracts the region-specific prognostic information in PT and MLN regions. Extensive experiments on the public dataset of HEad and neCK TumOR segmentation and outcome prediction challenge (HECKTOR 2022) demonstrate that our XSurv outperforms state-of-the-art survival prediction methods.
The University of Sydney（悉尼大学）；Shanghai Jiao Tong University（上海交通大学）
Outcome prediction is crucial for head and neck cancer patients as it can provide prognostic information for early treatment planning. Radiomics methods have been widely used for outcome prediction from medical images. However, these methods are limited by their reliance on intractable manual segmentation of tumor regions. Recently, deep learning methods have been proposed to perform end-to-end outcome prediction so as to remove the reliance on manual segmentation. Unfortunately, without segmentation masks, these methods will take the whole image as input, such that makes them difficult to foc…
In medical imaging, radiological scans of different modalities serve to enhance different sets of features for clinical diagnosis and treatment planning. This variety enriches the source information that could be used for outcome prediction. Deep learning methods are particularly well-suited for feature extraction from high-dimensional inputs such as images. In this work, we apply a CNN classification network augmented with a FCN preprocessor sub-network to a public TCIA head and neck cancer dataset. The training goal is survival prediction of radiotherapy cases based on pre-treatment FDG PET-…
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
When oncologists estimate cancer patient survival, they rely on multimodal data. Even though some multimodal deep learning methods have been proposed in the literature, the majority rely on having two or more independent networks that share knowledge at a later stage in the overall model. On the other hand, oncologists do not do this in their analysis but rather fuse the information in their brain from multiple sources such as medical images and patient history. This work proposes a deep learning method that mimics oncologists' analytical behavior when quantifying cancer and estimating patient…
University of Warwick（华威大学）；Shaukat Khanum Memorial Cancer Hospital Research Centre（肖卡特·汗姆纪念癌症医院研究中心）；University of Birmingham（伯明翰大学）；Newcastle University（纽卡斯尔大学）；University of Sheffield（谢菲尔德大学）；The Alan Turing Institute（阿兰·图灵研究所）；University Hospitals Coventry & Warwickshire NHS Trust（考文垂与华威郡大学医院NHS信托）
The infiltration of T-lymphocytes in the stroma and tumour is an indication of an effective immune response against the tumour, resulting in better survival. In this study, our aim is to explore the prognostic significance of tumour-associated stroma infiltrating lymphocytes (TASILs) in head and neck squamous cell carcinoma (HNSCC) through an AI based automated method. A deep learning based automated method was employed to segment tumour, stroma and lymphocytes in digitally scanned whole slide images of HNSCC tissue slides. The spatial patterns of lymphocytes and tumour-associated stroma were …
HES-SO Valais-Wallis University of Applied Sciences and Arts Western Switzerland（瑞士西部应用科学与艺术大学瓦莱州分校）；Centre Hospitalier Universitaire Vaudois (CHUV)（沃州大学中心医院）；Centre Hospitalier Universitaire de Poitiers (CHUP)（普瓦捷大学中心医院）；Univ Brest（布雷斯特大学）；INSERM（法国国家健康与医学研究院）；LaTIM（医学信息处理实验室）；Cleveland Clinic Foundation（克利夫兰医学中心基金会）；University of Sherbrooke（舍布鲁克大学）
This paper presents an overview of the second edition of the HEad and neCK TumOR (HECKTOR) challenge, organized as a satellite event of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2021. The challenge is composed of three tasks related to the automatic analysis of PET/CT images for patients with Head and Neck cancer (H&N), focusing on the oropharynx region. Task 1 is the automatic segmentation of H&N primary Gross Tumor Volume (GTVt) in FDG-PET/CT images. Task 2 is the automatic prediction of Progression Free Survival (PFS) from the s…
A Gradient Mapping Guided Explainable Deep Neural Network for Extracapsular Extension Identification in 3D Head and Neck Cancer Computed Tomography Images
Department of Industrial and Systems Engineering, Mississippi State University, Mississippi State, MS, 39762, USA（工业与系统工程系，密苏里州立大学）；Department of Radiation Oncology, University of Mississippi Medical Center, Jackson, MS, 39216, USA（放射肿瘤学系，密苏里大学医学中心）
Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosis
Division of Computational Pathology, Department of Pathology and Laboratory Medicine, Indiana University School of Medicine（计算病理学部，病理学与实验室医学部，印第安纳大学医学院）；IU Melvin and Bren Simon Comprehensive Cancer Center（印第安纳大学Melvin和Bren Simon综合癌症中心）；Departments of Biostatistics and Health Data Science（生物统计学与健康数据科学部）；Radiology and Imaging Sciences（放射学与影像科学部）；Neurological Surgery（神经外科）；Indiana University School of Medicine（印第安纳大学医学院）；Department of Computer Science, Luddy School of Informatics, Computing, and Engineering（计算机科学部，Luddy信息、计算与工程学院）
University of Toronto（多伦多大学）；Princess Margaret Cancer Centre（玛格丽特公主癌症中心）；TECHNA Institute（TECHNA研究所）
Accurate prognosis for an individual patient is a key component of precision oncology. Recent advances in machine learning have enabled the development of models using a wider range of data, including imaging. Radiomics aims to extract quantitative predictive and prognostic biomarkers from routine medical imaging, but evidence for computed tomography radiomics for prognosis remains inconclusive. We have conducted an institutional machine learning challenge to develop an accurate model for overall survival prediction in head and neck cancer using clinical data etxracted from electronic medical …
Segmentation of head and neck (H\&N) tumours and prediction of patient outcome are crucial for patient's disease diagnosis and treatment monitoring. Current developments of robust deep learning models are hindered by the lack of large multi-centre, multi-modal data with quality annotations. The MICCAI 2021 HEad and neCK TumOR (HECKTOR) segmentation and outcome prediction challenge creates a platform for comparing segmentation methods of the primary gross target volume on fluoro-deoxyglucose (FDG)-PET and Computed Tomography images and prediction of progression-free survival in H\&N oropharynge…
Southern Medical University（南方医科大学）；LaTIM（拉蒂姆实验室）；INSERM（法国国家健康与医学研究院）；Univ Brest（布雷斯特大学）；Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院）
Automatic segmentation of head and neck cancer (HNC) tumors and lymph nodes plays a crucial role in the optimization treatment strategy and prognosis analysis. This study aims to employ nnU-Net for automatic segmentation and radiomics for recurrence-free survival (RFS) prediction using pretreatment PET/CT images in multi-center HNC cohort. A multi-center HNC dataset with 883 patients (524 patients for training, 359 for testing) was provided in HECKTOR 2022. A bounding box of the extended oropharyngeal region was retrieved for each patient with fixed size of 224 x 224 x 224 $mm^{3}$. Then 3D nn…
University of British Columbia（不列颠哥伦比亚大学）；BC Cancer Research Institute（不列颠哥伦比亚癌症研究所）；Geneva University Hospital（日内瓦大学医院）
Radiomics features extract quantitative information from medical images, towards the derivation of biomarkers for clinical tasks, such as diagnosis, prognosis, or treatment response assessment. Different image discretization parameters (e.g. bin number or size), convolutional filters, segmentation perturbation, or multi-modality fusion levels can be used to generate radiomics features and ultimately signatures. Commonly, only one set of parameters is used; resulting in only one value or flavour for a given RF. We propose tensor radiomics (TR) where tensors of features calculated with multiple …
SimulaMet（SimulaMet研究院）；Oslo Metropolitan University（奥斯陆城市大学）；UiT The Arctic University of Norway（挪威北极大学）
Head and neck cancers are the fifth most common cancer worldwide, and recently, analysis of Positron Emission Tomography (PET) and Computed Tomography (CT) images has been proposed to identify patients with a prognosis. Even though the results look promising, more research is needed to further validate and improve the results. This paper presents the work done by team MLC for the 2022 version of the HECKTOR grand challenge held at MICCAI 2022. For Task 1, the automatic segmentation task, our approach was, in contrast to earlier solutions using 3D segmentation, to keep it as simple as possible …
Symb-xMIL: Symbolic Explanations for Multiple Instance Learning in Digital Pathology
Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）；Machine Learning Group, Technische Universität Berlin（柏林技术大学机器学习组）；Institute of Pathology, Charité Universitätsmedizin（查理研究所病理学部）；Berlin Institute of Health at Charité – Universitätsmedizin Berlin, BIH Biomedical Innovation Academy, BIH Charité Digital Clinician Scientist Program（柏林查理医学研究院健康研究所、BIH生物医学创新学院、BIH查理数字临床科学家项目）；Institute of Pathology, Ludwig Maximilian University of Munich（慕尼黑路德维希-马克西米利安大学病理学部）；Division of Translational Medical Oncology, DKFZ（转化医学肿瘤学部，德国有机化学研究所）；German Cancer Consortium (DKTK), partner site Munich, a partnership between DKFZ and Ludwig-Maximilians-Universität München (LMU)（德国癌症联盟（DKTK），慕尼黑合作伙伴站点，由DKFZ和路德维希-马克西米利安-慕尼黑大学（LMU）组成）；Department of Artificial Intelligence, Korea University（韩国大学人工智能系）；Max-Planck Institute for Informatics, Saarbrücken, Germany（马克斯·普朗克信息学院，萨尔布吕肯，德国）；Department of Chemistry, Chemical Physics Theory Group, University of Toronto（多伦多大学化学系，化学物理理论组）；Vector Institute for Artificial Intelligence, Toronto, Canada（多伦多人工智能矢量研究所）；Acceleration Consortium, University of Toronto（多伦多大学加速联盟）
University of Toronto（多伦多大学）；Princess Margaret Cancer Centre（玛格丽特公主癌症中心）
Predicting outcomes, such as survival or metastasis for individual cancer patients is a crucial component of precision oncology. Machine learning (ML) offers a promising way to exploit rich multi-modal data, including clinical information and imaging to learn predictors of disease trajectory and help inform clinical decision making. In this paper, we present a novel graph-based approach to incorporate imaging characteristics of existing cancer spread to local lymph nodes (LNs) as well as their connectivity patterns in a prognostic ML model. We trained an edge-gated Graph Convolutional Network …
Hamburg University of Technology（汉堡工业大学）；Carl von Ossietzky University Oldenburg（奥尔登堡大学）；University Medical Center Hamburg-Eppendorf（汉堡-埃彭多夫大学医学中心）；University of Bonn（波恩大学）；Klinikum Braunschweig（不伦瑞克医院）
Early detection of cancerous tissue is crucial for long-term patient survival. In the head and neck region, a typical diagnostic procedure is an endoscopic intervention where a medical expert manually assesses tissue using RGB camera images. While healthy and tumor regions are generally easier to distinguish, differentiating benign and malignant tumors is very challenging. This requires an invasive biopsy, followed by histological evaluation for diagnosis. Also, during tumor resection, tumor margins need to be verified by histological analysis. To avoid unnecessary tissue resection, a non-inva…
