Planar 3D Transfer Learning for End to End Unimodal MRI Unbalanced Data Segmentation
Brno University of Technology（布尔诺理工大学）；University of Las Palmas de Gran Canaria（大加那利岛拉斯帕尔马斯大学）；Masaryk University（马萨里克大学）；University Hospital Brno（布尔诺大学医院）
Martin Kolarik, Radim Burget, Carlos M. Travieso-Gonzalez, Jan Kocica
We present a novel approach of 2D to 3D transfer learning based on mapping pre-trained 2D convolutional neural network weights into planar 3D kernels. The method is validated by the proposed planar 3D res-u-net network with encoder transferred from the 2D VGG-16, which is applied for a single-stage unbalanced 3D image data segmentation. In particular, we evaluate the method on the MICCAI 2016 MS lesion segmentation challenge dataset utilizing solely fluid-attenuated inversion recovery (FLAIR) sequence without brain extraction for training and inference to simulate real medical praxis. The planar 3D res-u-net network performed the best both in sensitivity and Dice score amongst end to end methods processing raw MRI scans and achieved comparable Dice score to a state-of-the-art unimodal not end to end approach. Complete source code was released under the open-source license, and this paper complies with the Machine learning reproducibility checklist. By implementing practical transfer learning for 3D data representation, we could segment heavily unbalanced data without selective sampling and achieved more reliable results using less training data in a single modality. From a medical perspective, the unimodal approach gives an advantage in real praxis as it does not require co-registration nor additional scanning time during an examination. Although modern medical imaging methods capture high-resolution 3D anatomy scans suitable for computer-aided detection system processing, deployment of automatic systems for interpretation of radiology imaging is still rather theoretical in many medical areas. Our work aims to bridge the gap by offering a solution for partial research questions.
CREATIS（CREATIS实验室）；Université Lyon 1（里昂第一大学）；INSA-Lyon（里昂国立应用科学学院）；Hospices Civils de Lyon（里昂民用医院）；CarMeN（CarMeN实验室）；LARIS（LARIS实验室）；Universite d’Angers（昂热大学）
Patient outcome prediction is critical in management of ischemic stroke. In this paper, a novel machine learning model is proposed for stroke outcome prediction using multimodal Magnetic Resonance Imaging (MRI). The proposed model consists of two serial levels of Autoencoders (AEs), where different AEs at level 1 are used for learning unimodal features from different MRI modalities and a AE at level 2 is used to combine the unimodal features into compressed multimodal features. The sequences of multimodal features of a given patient are then used by an LSTM network for predicting outcome score…
VocSegMRI: Multimodal Learning for Precise Vocal Tract Segmentation in Real-time MRI
Friedrich-Alexander-Universität Erlangen-Nürnberg（弗里德里希-亚历山大-埃朗根-纽伦堡大学）；University of Maryland School of Dentistry（马里兰大学牙科学院）；Harvard Medical School/Massachusetts General Hospital（哈佛医学院/麻省总医院）；Universidad de Antioquia UdeA（安蒂奥基亚大学 UdeA）；Johns Hopkins University（约翰霍普金斯大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
Improving Quality Control Of MRI Images Using Synthetic Motion Data
Mass General Brigham（麻省总医院布列根医疗系统）；Washington University（华盛顿大学）；UCLA（加州大学洛杉矶分校）；Orygen；Mt Sinai（西奈山医院）；Northwell Health（诺斯韦尔医疗）；Yale（耶鲁大学）；University of Calgary（卡尔加里大学）；Mila（米拉人工智能研究所）
Universitat Politècnica de Catalunya. BarcelonaTech（加泰罗尼亚理工大学·巴塞罗那理工）
Automation of brain tumor segmentation in 3D magnetic resonance images (MRIs) is key to assess the diagnostic and treatment of the disease. In recent years, convolutional neural networks (CNNs) have shown improved results in the task. However, high memory consumption is still a problem in 3D-CNNs. Moreover, most methods do not include uncertainty information, which is especially critical in medical diagnosis. This work studies 3D encoder-decoder architectures trained with patch-based techniques to reduce memory consumption and decrease the effect of unbalanced data. The different trained model…
A-QCF-Net: An Adaptive Quaternion Cross-Fusion Network for Multimodal Liver Tumor Segmentation from Unpaired Datasets
University College of Engineering, Bharathidasan Institute of Technology Campus, Anna University（安娜大学工程学院，巴拉特拉桑理工学院校区，安娜大学）；National Institute of Technology（国家理工学院）
Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI
Harvard Medical School / Massachusetts General Hospital（哈佛医学院/麻省总医院）；Institute for Information Processing, Leibniz University Hannover（汉诺威莱布尼茨信息处理研究所）；GITA Lab, Facultad de Ingeniería. Universidad de Antioquia UdeA（安提奥基亚大学工程学院GITA实验室）
Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables …
Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcare
University of Aberdeen（阿伯丁大学）；NepAl Applied Mathematics and Informatics Institute for research, Nepal（尼泊尔应用数学与信息学研究所）；University of Lausanne（洛桑大学）；West Virginia University（西弗吉尼亚大学）；University of Verona（威尼斯大学）；University College London（伦敦大学学院）
University College London（伦敦大学学院）；University of Côte d’Azur（蔚蓝海岸大学）；Inria Sophia Antipolis（法国国家信息与自动化研究所索菲亚·安提波利斯中心）
We introduce Disease Knowledge Transfer (DKT), a novel technique for transferring biomarker information between related neurodegenerative diseases. DKT infers robust multimodal biomarker trajectories in rare neurodegenerative diseases even when only limited, unimodal data is available, by transferring information from larger multimodal datasets from common neurodegenerative diseases. DKT is a joint-disease generative model of biomarker progressions, which exploits biomarker relationships that are shared across diseases. Our proposed method allows, for the first time, the estimation of plausibl…
Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）；Leibniz Institute for Prevention Research and Epidemiology – BIPS（莱比锡预防研究与流行病学研究所 – BIPS）；Faculty of Mathematics and Computer Science, University of Bremen（不莱梅大学数学与计算机科学学院）；Faculty of Physics and Electrical Engineering, University of Bremen（不莱梅大学物理与电气工程学院）
Magnetic resonance imaging (MRI) is being increasingly utilized to assess, diagnose, and plan treatment for a variety of diseases. The ability to visualize tissue in varied contrasts in the form of MR pulse sequences in a single scan provides valuable insights to physicians, as well as enabling automated systems performing downstream analysis. However many issues like prohibitive scan time, image corruption, different acquisition protocols, or allergies to certain contrast materials may hinder the process of acquiring multiple sequences for a patient. This poses challenges to both physicians a…
Universitat Politècnica de Catalunya. BarcelonaTech（加泰罗尼亚理工大学）
Automation of brain tumors in 3D magnetic resonance images (MRIs) is key to assess the diagnostic and treatment of the disease. In recent years, convolutional neural networks (CNNs) have shown improved results in the task. However, high memory consumption is still a problem in 3D-CNNs. Moreover, most methods do not include uncertainty information, which is specially critical in medical diagnosis. This work proposes a 3D encoder-decoder architecture, based on V-Net \cite{vnet} which is trained with patching techniques to reduce memory consumption and decrease the effect of unbalanced data. We a…
School of Computer Science, University of Nottingham（诺丁汉大学计算机学院）；School of Medicine, University of Nottingham（诺丁汉大学医学院）；Sir Peter Mansfield Imaging Centre, University of Nottingham（诺丁汉大学彼得·曼斯菲尔德爵士成像中心）
In this paper, we present a generic deep convolutional neural network (DCNN) for multi-class image segmentation. It is based on a well-established supervised end-to-end DCNN model, known as U-net. U-net is firstly modified by adding widely used batch normalization and residual block (named as BRU-net) to improve the efficiency of model training. Based on BRU-net, we further introduce a dynamically weighted cross-entropy loss function. The weighting scheme is calculated based on the pixel-wise prediction accuracy during the training process. Assigning higher weights to pixels with lower segment…
