To study neurodegenerative diseases, longitudinal studies are carried on volunteer patients. During a time span of several months to several years, they go through regular medical visits to acquire data from different modalities, such as biological samples, cognitive tests, structural and functional imaging. These variables are heterogeneous but they all depend on the patient's health condition, meaning that there are possibly unknown relationships between all modalities. Some information may be specific to some modalities, others may be complementary, and others may be redundant. Some data may also be missing. In this work we present a neural network architecture for multimodal learning, able to use imaging and clinical data from two time points to predict the evolution of a neurodegenerative disease, and robust to missing values. Our multimodal network achieves 92.5\% accuracy and an AUC score of 0.978 over a test set of 57 subjects. We also show the superiority of the multimodal architecture, for up to 37.5\% of missing values in test set subjects' clinical measurements, compared to a model using only the clinical modality.
Indian Institute of Technology Jodhpur（印度理工学院焦特布尔分校）；Indian Institute of Technology Mandi（印度理工学院曼迪分校）
In this paper, we propose a novel deep learning architecture combining stacked Bi-directional LSTM and LSTMs with the Siamese network architecture for segmentation of brain fibers, obtained from tractography data, into anatomically meaningful clusters. The proposed network learns the structural difference between fibers of different classes, which enables it to classify fibers with high accuracy. Importantly, capturing such deep inter and intra class structural relationship also ensures that the segmentation is robust to relative rotation among test and training data, hence can be used with un…
Precise identification of mouse brain microscopy images is a crucial first step when anatomical structures in the mouse brain are to be registered to a reference atlas. Practitioners usually rely on manual comparison of images or tools that assume the presence of complete images. This work explores Siamese Networks as the method for finding corresponding 2D reference atlas plates for given partial 2D mouse brain images. Siamese networks are a class of convolutional neural networks (CNNs) that use weight-shared paths to obtain low dimensional embeddings of pairs of input images. The corresponde…
Improving Cytoarchitectonic Segmentation of Human Brain Areas with Self-supervised Siamese Networks
Institute of Neuroscience and Medicine INM-1, Forschungszentrum Jülich（于利希研究中心神经科学与医学研究所（INM-1））；C. and O. Vogt Institute of Brain Research, Heinrich-Heine University Düsseldorf（杜塞尔多夫海因里希·海涅大学C.和O.福格特脑研究所）；Institute of Computer Science, Heinrich-Heine University Düsseldorf（杜塞尔多夫海因里希·海涅大学计算机科学研究所）
Centre for Neuroscience, Indian Institute of Science（印度科学学院神经科学中心）
The Algonauts challenge is about predicting the object representations in the form of Representational Dissimilarity Matrices (RDMS) derived from visual brain regions. We used a customized deep learning model using the concept of Siamese networks and group convolutions to predict neural distances corresponding to a pair of images. Training data was best explained by distances computed over the last layer.
Prediction of the cognitive evolution of a person susceptible to develop a neurodegenerative disorder is crucial to provide an appropriate treatment as soon as possible. In this paper we propose a 3D siamese network designed to extract features from whole-brain 3D MRI images. We show that it is possible to extract meaningful features using convolution layers, reducing the need of classical image processing operations such as segmentation or pre-computing features such as cortical thickness. To lead this study we used the Alzheimer's Disease Neuroimaging Initiative (ADNI), a public data base of…
Exploring Siamese Networks in Self-Supervised Fast MRI Reconstruction
Stanford University School of Medicine（斯坦福大学医学院）；Xiamen University（厦门大学）；School of Electronic Science and Engineering, Xiamen University（厦门大学电子科学与工程学院）；School of Informatics, Xiamen University（厦门大学信息学院）；Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能研究院）
Shanghai University（上海大学）；JD AI Research（京东人工智能研究院）；NLPR, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；Institute of Automation, Chinese Academy of Science（中国科学院自动化研究所）
Most existing public face datasets, such as MS-Celeb-1M and VGGFace2, provide abundant information in both breadth (large number of IDs) and depth (sufficient number of samples) for training. However, in many real-world scenarios of face recognition, the training dataset is limited in depth, i.e. only two face images are available for each ID. $\textit{We define this situation as Shallow Face Learning, and find it problematic with existing training methods.}$ Unlike deep face data, the shallow face data lacks intra-class diversity. As such, it can lead to collapse of feature dimension and cons…
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Intel Labs（英特尔研究院）；Rutgers University（罗格斯大学）；Yale University（耶鲁大学）
Learning a similarity metric has gained much attention recently, where the goal is to learn a function that maps input patterns to a target space while preserving the semantic distance in the input space. While most related work focused on images, we focus instead on learning a similarity metric for neuroimages, such as fMRI and DTI images. We propose an end-to-end similarity learning framework called Higher-order Siamese GCN for multi-subject fMRI data analysis. The proposed framework learns the brain network representations via a supervised metric-based approach with siamese neural networks …
Univ Lyon（里昂大学）；INSA‐Lyon（里昂国立应用科学学院）；Université Claude Bernard Lyon 1（里昂第一大学）；UJM-Saint Etienne（圣艾蒂安让·莫奈大学）；CNRS（法国国家科学研究中心）；Inserm（法国国家健康与医学研究院）；CREATIS UMR 5220（CREATIS联合研究单位5220）；U1206（U1206研究单位）
Computer aided diagnosis (CAD) systems are designed to assist clinicians in various tasks, including highlighting abnormal regions in a medical image. A common approach consists in training a voxel-level binary classifier on a set of feature vectors extracted from normal and pathological areas in patients' scans. However, many pathologies (such as epilepsy) are characterized by lesions that may be located anywhere in the brain, have various shapes, sizes and texture. An adequate representation of such a heterogeneity requires a significant amount of annotated data which is a major issue in the…
University of Catania（卡塔尼亚大学）；University of Central Florida（中佛罗里达大学）
This work presents a novel method of exploring human brain-visual representations, with a view towards replicating these processes in machines. The core idea is to learn plausible computational and biological representations by correlating human neural activity and natural images. Thus, we first propose a model, EEG-ChannelNet, to learn a brain manifold for EEG classification. After verifying that visual information can be extracted from EEG data, we introduce a multimodal approach that uses deep image and EEG encoders, trained in a siamese configuration, for learning a joint manifold that max…
Instituto Superior Tecnico（高等技术学院）；Universitat Politecnica de Catalunya（加泰罗尼亚理工大学）
The increasing access to brain signal data using electroencephalography creates new opportunities to study electrophysiological brain activity and perform ambulatory diagnoses of neuronal diseases. This work proposes a pairwise distance learning approach for Schizophrenia classification relying on the spectral properties of the signal. Given the limited number of observations (i.e. the case and/or control individuals) in clinical trials, we propose a Siamese neural network architecture to learn a discriminative feature space from pairwise combinations of observations per channel. In this way, …
Early Autism Diagnosis based on Path Signature and Siamese Unsupervised Feature Compressor
University of Pennsylvania（宾夕法尼亚大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；The Affiliated Shenzhen School of Guangdong Experimental High School（广东实验中学附属深圳学校）；Pazhou Lab（琶洲实验室）
A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Experts
Imperial College London（帝国理工学院）；UK Dementia Research Institute（英国痴呆症研究所）；Beijing Institute of Technology（北京理工大学）；Capital Medical University（首都医科大学）；Beijing Tiantan Hospital（北京天坛医院）
Hierarchical Brain Structure Modeling for Predicting Genotype of Glioma
College of Medicine and Biological Information Engineering, Northeastern University, Liaoning, China（医学与生物信息工程学院，东北大学，辽宁，中国）；School of Science and Engineering, University of Dundee, Dundee, UK（科学与工程学院，邓迪大学，邓迪，英国）；School of Medicine, University of Dundee, Dundee, UK（医学院，邓迪大学，邓迪，英国）；Department of Applied Maths and Theoretical Physics, University of Cambridge, Cambridge, UK（应用数学与理论物理系，剑桥大学，剑桥，英国）
Delft University of Technology（代尔夫特理工大学）；University of Copenhagen（哥本哈根大学）；University Medical Center Utrecht（乌得勒支大学医学中心）；Netherlands eScience Center（荷兰电子科学中心）
Voxelwise classification approaches are popular and effective methods for tissue quantification in brain magnetic resonance imaging (MRI) scans. However, generalization of these approaches is hampered by large differences between sets of MRI scans such as differences in field strength, vendor or acquisition protocols. Due to this acquisition related variation, classifiers trained on data from a specific scanner fail or under-perform when applied to data that was acquired differently. In order to address this lack of generalization, we propose a Siamese neural network (MRAI-net) to learn a repr…
Objective: Epilepsy is one of the most prevalent neurological diseases among humans and can lead to severe brain injuries, strokes, and brain tumors. Early detection of seizures can help to mitigate injuries, and can be used to aid the treatment of patients with epilepsy. The purpose of a seizure prediction system is to successfully identify the pre-ictal brain stage, which occurs before a seizure event. Patient-independent seizure prediction models are designed to offer accurate performance across multiple subjects within a dataset, and have been identified as a real-world solution to the sei…
Increasing effort in brain image analysis has been dedicated to early diagnosis of Alzheimer's disease (AD) based on neuroimaging data. Most existing studies have been focusing on binary classification problems, e.g., distinguishing AD patients from normal control (NC) elderly or mild cognitive impairment (MCI) individuals from NC elderly. However, identifying individuals with AD and MCI, especially MCI individuals who will convert to AD (progressive MCI, pMCI), in a single setting, is needed to achieve the goal of early diagnosis of AD. In this paper, we propose a deep ordinal ranking model f…
