Single-subject mapping of resting-state brain functional activity to non-imaging phenotypes is a major goal of neuroimaging. The large majority of learning approaches applied today rely either on static representations or on short-term temporal correlations. This is at odds with the nature of brain activity which is dynamic and exhibit both short- and long-range dependencies. Further, new sophisticated deep learning approaches have been developed and validated on single tasks/datasets. The application of these models for the study of a different targets typically require exhaustive hyperparameter search, model engineering and trial and error to obtain competitive results with simpler linear models. This in turn limit their adoption and hinder fair benchmarking in a rapidly developing area of research. To this end, we propose fMRI-S4; a versatile deep learning model for the classification of phenotypes and psychiatric disorders from the timecourses of resting-state functional magnetic resonance imaging scans. fMRI-S4 capture short- and long- range temporal dependencies in the signal using 1D convolutions and the recently introduced state-space models S4. The proposed architecture is lightweight, sample-efficient and robust across tasks/datasets. We validate fMRI-S4 on the tasks of diagnosing major depressive disorder (MDD), autism spectrum disorder (ASD) and sex classifcation on three multi-site rs-fMRI datasets. We show that fMRI-S4 can outperform existing methods on all three tasks and can be trained as a plug&play model without special hyperpararameter tuning for each setting
K. N. Toosi University of Technology（K.N.托西理工大学）；Ferdowsi University of Mashhad（马什哈德菲尔多西大学）；Islamic Azad University（伊斯兰阿扎德大学）；Deakin University（迪肯大学）；Universidad de Granada（格拉纳达大学）；Ngee Ann Polytechnic（义安理工学院）；Asia University（亚洲大学）；Singapore University of Social Sciences（新加坡社会科学大学）
Nowadays, many people worldwide suffer from brain disorders, and their health is in danger. So far, numerous methods have been proposed for the diagnosis of Schizophrenia (SZ) and attention deficit hyperactivity disorder (ADHD), among which functional magnetic resonance imaging (fMRI) modalities are known as a popular method among physicians. This paper presents an SZ and ADHD intelligent detection method of resting-state fMRI (rs-fMRI) modality using a new deep learning method. The University of California Los Angeles dataset, which contains the rs-fMRI modalities of SZ and ADHD patients, has…
An Attention-Based Framework for Alzheimers Disease Classification Using Resting-State fMRI
School of Medical Science and Technology（医学科学与技术学院）；Dr B C Roy Multi Speciality Medical Research Centre（B C Roy多专科医学研究中心）；Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）；Indian Institute of Technology Gandhinagar（印度理工学院甘地纳格尔分校）
Aspect-level sentiment classification aims to distinguish the sentiment polarities over one or more aspect terms in a sentence. Existing approaches mostly model different aspects in one sentence independently, which ignore the sentiment dependencies between different aspects. However, we find such dependency information between different aspects can bring additional valuable information. In this paper, we propose a novel aspect-level sentiment classification model based on graph convolutional networks (GCN) which can effectively capture the sentiment dependencies between multi-aspects in one s…
STARFormer: A Novel Spatio-Temporal Aggregation Reorganization Transformer of FMRI for Brain Disorder Diagnosis
Laboratory of Digital Image and Intelligent Computation, College of Information Engineering, Shanghai Maritime University（数字图像与智能计算实验室，信息工程学院，上海 Maritime 大学）；Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University（中英双语研究部，香港理工大学）；Department of Neurology, Affiliated Lianyungang Hospital of Xuzhou Medical University（神经科，徐州医学院附属连云港医院）
针对 fMRI 脑疾病分类忽略时空依赖的问题，提出 STARFormer，通过 ROI 空间重组、时间特征重组和时空融合三个模块，在 ASD 和 ADHD 数据集上达到最先进性能。
Columbia University（哥伦比亚大学）；New York State Psychiatric Institute（纽约州精神病学研究所）
The goal of emotional brain state classification on functional MRI (fMRI) data is to recognize brain activity patterns related to specific emotion tasks performed by subjects during an experiment. Distinguishing emotional brain states from other brain states using fMRI data has proven to be challenging due to two factors: a difficulty to generate fast yet accurate predictions in short time frames, and a difficulty to extract emotion features which generalize to unseen subjects. To address these challenges, we conducted an experiment in which 22 subjects viewed pictures designed to stimulate ei…
Johns Hopkins University（约翰斯·霍普金斯大学）；Microsoft（微软公司）；Johns Hopkins School of Medicine（约翰斯·霍普金斯医学院）；Malone Center for Engineering in Healthcare（马龙医疗保健工程中心）
National Institute of Technology Tiruchirappalli（蒂鲁吉拉伯利国家理工学院）；Sharif University of Technology（谢里夫理工大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Multi-modal data comprising imaging (MRI, fMRI, PET, etc.) and non-imaging (clinical test, demographics, etc.) data can be collected together and used for disease prediction. Such diverse data gives complementary information about the patientś condition to make an informed diagnosis. A model capable of leveraging the individuality of each multi-modal data is required for better disease prediction. We propose a graph convolution based deep model which takes into account the distinctiveness of each element of the multi-modal data. We incorporate a novel self-attention layer, which weights every …
Wayne State University（韦恩州立大学）；New York University（纽约大学）
Understanding human fetal neurodevelopment is of great clinical importance as abnormal development is linked to adverse neuropsychiatric outcomes after birth. Recent advances in functional Magnetic Resonance Imaging (fMRI) have provided new insight into development of the human brain before birth, but these studies have predominately focused on brain functional connectivity (i.e. Fisher z-score), which requires manual processing steps for feature extraction from fMRI images. Deep learning approaches (i.e., Convolutional Neural Networks) have achieved remarkable success on learning directly fro…
Soochow University（苏州大学）；University of North Carolina at Greensboro（北卡罗来纳大学格林斯伯勒分校）；The Hong Kong Polytechnic University（香港理工大学）
Traffic forecasting is one of the most fundamental problems in transportation science and artificial intelligence. The key challenge is to effectively model complex spatial-temporal dependencies and correlations in modern traffic data. Existing methods, however, cannot accurately model both long-term and short-term temporal correlations simultaneously, limiting their expressive power on complex spatial-temporal patterns. In this paper, we propose a novel spatial-temporal neural network framework: Attention-based Spatial-Temporal Graph Convolutional Recurrent Network (ASTGCRN), which consists o…
Graph Signal Processing is a promising framework to manipulate brain signals as it allows to encompass the spatial dependencies between the activity in regions of interest in the brain. In this work, we are interested in better understanding what are the graph frequencies that are the most useful to decode fMRI signals. To this end, we introduce a deep learning architecture and adapt a pruning methodology to automatically identify such frequencies. We experiment with various datasets, architectures and graphs, and show that low graph frequencies are consistently identified as the most importan…
The Blood-Oxygen-Level-Dependent (BOLD) signal of resting-state fMRI (rs-fMRI) records the temporal dynamics of intrinsic functional networks in the brain. However, existing deep learning methods applied to rs-fMRI either neglect the functional dependency between different brain regions in a network or discard the information in the temporal dynamics of brain activity. To overcome those shortcomings, we propose to formulate functional connectivity networks within the context of spatio-temporal graphs. We train a spatio-temporal graph convolutional network (ST-GCN) on short sub-sequences of the…
This work proposes an algorithm for fMRI data analysis for the classification of ADHD disorders. There have been several breakthroughs in the analysis of fMRI via 3D convolutional neural networks (CNNs). With these new techniques it is possible to preserve the 3D spatial data of fMRI data. Additionally there have been recent advances in the use of 3D generative adversarial neural networks (GANs) for the generation of normal MRI data. This work utilizes multi modal 3D CNNs with data augmentation from 3D GAN for ADHD prediction from fMRI. By leveraging a 3D-GAN it would be possible to use deepfa…
Finding the most effective way to aggregate multi-subject fMRI data is a long-standing and challenging problem. It is of increasing interest in contemporary fMRI studies of human cognition due to the scarcity of data per subject and the variability of brain anatomy and functional response across subjects. Recent work on latent factor models shows promising results in this task but this approach does not preserve spatial locality in the brain. We examine two ways to combine the ideas of a factor model and a searchlight based analysis to aggregate multi-subject fMRI data while preserving spatial…
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Research Centre for Natural Sciences, Hungarian Academy of Sciences（匈牙利科学院自然科学研究中心）
Machine learning techniques have become increasingly popular in the field of resting state fMRI (functional magnetic resonance imaging) network based classification. However, the application of convolutional networks has been proposed only very recently and has remained largely unexplored. In this paper we describe a convolutional neural network architecture for functional connectome classification called connectome-convolutional neural network (CCNN). Our results on simulated datasets and a publicly available dataset for amnestic mild cognitive impairment classification demonstrate that our C…
Deep learning methods are increasingly being used with neuroimaging data like structural and function magnetic resonance imaging (MRI) to predict the diagnosis of neuropsychiatric and neurological disorders. For psychiatric disorders in particular, it is believed that one of the most promising modality is the resting-state functional MRI (rsfMRI), which captures the intrinsic connectivity between regions in the brain. Because rsfMRI data points are inherently high-dimensional (~1M), it is impossible to process the entire input in its raw form. In this paper, we propose a very simple transforma…
University of Electronic Science and Technology of China（电子科技大学）；East China Normal University（华东师范大学）；University of British Columbia（不列颠哥伦比亚大学）
In this work, we focus on the challenging task, neuro-disease classification, using functional magnetic resonance imaging (fMRI). In population graph-based disease analysis, graph convolutional neural networks (GCNs) have achieved remarkable success. However, these achievements are inseparable from abundant labeled data and sensitive to spurious signals. To improve fMRI representation learning and classification under a label-efficient setting, we propose a novel and theory-driven self-supervised learning (SSL) framework on GCNs, namely Graph CCA for Temporal self-supervised learning on fMRI a…
Due to the low signal-to-noise ratio and limited resolution of functional MRI data, and the high complexity of natural images, reconstructing a visual stimulus from human brain fMRI measurements is a challenging task. In this work, we propose a novel approach for this task, which we call Cortex2Image, to decode visual stimuli with high semantic fidelity and rich fine-grained detail. In particular, we train a surface-based convolutional network model that maps from brain response to semantic image features first (Cortex2Semantic). We then combine this model with a high-quality image generator (…
NeuroMambaLLM: Dynamic Graph Learning of fMRI Functional Connectivity in Autistic Brains Using Mamba and Language Model Reasoning
Department of Electrical and Computer Engineering, McMaster University（麦基尔大学电气与计算机工程系）；Department of Biomedical Engineering, University of Southern California（南加州大学生物医学工程系）；Department of Electrical and Computer Engineering, University of Rochester（罗切斯特大学电气与计算机工程系）；BIOSEN Group（BIOSEN集团）
Brain tumor segmentation is a challenging problem in medical image analysis. The endpoint is to generate the salient masks that accurately identify brain tumor regions in an fMRI screening. In this paper, we propose a novel attention gate (AG model) for brain tumor segmentation that utilizes both the edge detecting unit and the attention gated network to highlight and segment the salient regions from fMRI images. This feature enables us to eliminate the necessity of having to explicitly point towards the damaged area(external tissue localization) and classify(classification) as per classical c…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We present an empirical evaluation of fMRI data augmentation via synthesis. For synthesis we use generative mod-els trained on real neuroimaging data to produce novel task-dependent functional brain images. Analyzed generative mod-els include classic approaches such as the Gaussian mixture model (GMM), and modern implicit generative models such as the generative adversarial network (GAN) and the variational auto-encoder (VAE). In particular, the proposed GAN and VAE models utilize 3-dimensional convolutions, which enables modeling of high-dimensional brain image tensors with structured spatial…
Direct Estimation of Pediatric Heart Rate Variability from BOLD-fMRI: A Machine Learning Approach Using Dynamic Connectivity
University of Calgary（卡尔加里大学）；Cumming School of Medicine, University of Calgary（卡尔加里大学卡明医学院）；Hotchkiss Brain Institute, Cumming School of Medicine, University of Calgary（卡尔加里大学卡明医学院霍奇基斯脑研究所）；University of New England（新英格兰大学）
Ensemble manifold based regularized multi-modal graph convolutional network for cognitive ability prediction
Tulane University（杜兰大学）；Xavier University of Louisiana（路易斯安那泽维尔大学）；Georgia State University（佐治亚州立大学）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）
University of Brescia（布雷西亚大学）；Tel Aviv Sourasky Medical Center（特拉维夫苏拉斯基医疗中心）；Maastricht University（马斯特里赫特大学）
Deep neural networks have been developed drawing inspiration from the brain visual pathway, implementing an end-to-end approach: from image data to video object classes. However building an fMRI decoder with the typical structure of Convolutional Neural Network (CNN), i.e. learning multiple level of representations, seems impractical due to lack of brain data. As a possible solution, this work presents the first hybrid fMRI and deep features decoding approach: collected fMRI and deep learnt representations of video object classes are linked together by means of Kernel Canonical Correlation Ana…
Amsterdam UMC（阿姆斯特丹大学医学中心）；University of Amsterdam（阿姆斯特丹大学）
The characterisation of the brain as a functional network in which the connections between brain regions are represented by correlation values across time series has been very popular in the last years. Although this representation has advanced our understanding of brain function, it represents a simplified model of brain connectivity that has a complex dynamic spatio-temporal nature. Oversimplification of the data may hinder the merits of applying advanced non-linear feature extraction algorithms. To this end, we propose a dynamic adaptive spatio-temporal graph convolution (DAST-GCN) model to…
Deep Learning Inductive Biases for fMRI Time Series Classification during Resting-state and Movie-watching
School of Electrical and Computer Engineering, University of Tehran（埃特瓦尔大学电气与计算机工程学院）；School of Cognitive Sciences, Institute for Research in Fundamental Sciences (IPM)（基础科学研究所（IPM）认知科学学院）
Simultaneous EEG-fMRI is a multi-modal neuroimaging technique that provides complementary spatial and temporal resolution for inferring a latent source space of neural activity. In this paper we address this inference problem within the framework of transcoding -- mapping from a specific encoding (modality) to a decoding (the latent source space) and then encoding the latent source space to the other modality. Specifically, we develop a symmetric method consisting of a cyclic convolutional transcoder that transcodes EEG to fMRI and vice versa. Without any prior knowledge of either the hemodyna…
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Intel Labs（英特尔研究院）；Rutgers University（罗格斯大学）；Yale University（耶鲁大学）
Learning a similarity metric has gained much attention recently, where the goal is to learn a function that maps input patterns to a target space while preserving the semantic distance in the input space. While most related work focused on images, we focus instead on learning a similarity metric for neuroimages, such as fMRI and DTI images. We propose an end-to-end similarity learning framework called Higher-order Siamese GCN for multi-subject fMRI data analysis. The proposed framework learns the brain network representations via a supervised metric-based approach with siamese neural networks …
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Graph neural networks (GNN) rely on graph operations that include neural network training for various graph related tasks. Recently, several attempts have been made to apply the GNNs to functional magnetic resonance image (fMRI) data. Despite recent progresses, a common limitation is its difficulty to explain the classification results in a neuroscientifically explainable way. Here, we develop a framework for analyzing the fMRI data using the Graph Isomorphism Network (GIN), which was recently proposed as a powerful GNN for graph classification. One of the important contributions of this paper…
Simultaneous EEG-fMRI is a multi-modal neuroimaging technique that provides complementary spatial and temporal resolution. Challenging has been developing principled and interpretable approaches for fusing the modalities, specifically approaches enabling inference of latent source spaces representative of neural activity. In this paper, we address this inference problem within the framework of transcoding -- mapping from a specific encoding (modality) to a decoding (the latent source space) and then encoding the latent source space to the other modality. Specifically, we develop a symmetric me…
National Tsing Hua University（国立清华大学）；MOST Joint Research Center for AI Technology and All Vista Healthcare（科技部人工智能技术与全视野医疗联合研究中心）；National Health Research Institutes（国家卫生研究院）
The ability to accurately detect onset of dementia is important in the treatment of the disease. Clinically, the diagnosis of Alzheimer Disease (AD) and Mild Cognitive Impairment (MCI) patients are based on an integrated assessment of psychological tests and brain imaging such as positron emission tomography (PET) and anatomical magnetic resonance imaging (MRI). In this work using two different datasets, we propose a behavior score-embedded encoder network (BSEN) that integrates regularly adminstrated psychological tests information into the encoding procedure of representing subject's resting…
Graph Autoencoders for Embedding Learning in Brain Networks and Major Depressive Disorder Identification
Monash University Malaysia（莫纳什大学马来西亚校区）；Vanderbilt University Medical Center（范德比尔特大学医学中心）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Brain-MGF: Multimodal Graph Fusion Network for EEG-fMRI Brain Connectivity Analysis Under Psilocybin
School of Information Technology, Monash University Malaysia（信息技术学院，墨尔本大学马来西亚分校）；Department of Data Science and AI, Faculty of Information Technology, Monash University（数据科学与人工智能系，信息科技学院，墨尔本大学）；Turner Institute for Brain and Mental Health, School of Psychological Sciences, Monash University（脑与心理健康研究所，心理学科学学院，墨尔本大学）
Interpretable Information-Decomposed Brain Graph Learning for fMRI-based Disease Diagnosis
School of Mathematics and Statistics, Shandong University（山东大学数学与统计学院）；Data Science Institute, Shandong University（山东大学数据科学研究院）；Academy of Mathematics and Systems Science, University of Chinese Academy of Sciences（中国科学院大学数学与系统科学研究院）
University of Cambridge（剑桥大学）；University of Granada（格拉纳达大学）
Deep learning models for MRI classification face two recurring problems: they are typically limited by low sample size, and are abstracted by their own complexity (the "black box problem"). In this paper, we train a convolutional neural network (CNN) with the largest multi-source, functional MRI (fMRI) connectomic dataset ever compiled, consisting of 43,858 datapoints. We apply this model to a cross-sectional comparison of autism (ASD) vs typically developing (TD) controls that has proved difficult to characterise with inferential statistics. To contextualise these findings, we additionally pe…
PUCRS（南里奥格兰德州天主教大学）；BraIns, Brain Institute of Rio Grande do Sul（南里奥格兰德州大脑研究所）
Brain imaging of mental health, neurodevelopmental and learning disorders has coupled with machine learning to identify patients based only on their brain activation, and ultimately identify features that generalize from smaller samples of data to larger ones. However, the success of machine learning classification algorithms on neurofunctional data has been limited to more homogeneous data sets of dozens of participants. More recently, larger brain imaging data sets have allowed for the application of deep learning techniques to classify brain states and clinical groups solely from neurofunct…
Learning representations that accurately capture long-range dependencies in sequential inputs -- including text, audio, and genomic data -- is a key problem in deep learning. Feed-forward convolutional models capture only feature interactions within finite receptive fields while recurrent architectures can be slow and difficult to train due to vanishing gradients. Here, we propose Temporal Feature-Wise Linear Modulation (TFiLM) -- a novel architectural component inspired by adaptive batch normalization and its extensions -- that uses a recurrent neural network to alter the activations of a con…
Texas Tech University（德克萨斯理工大学）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）
Deep learning with Convolutional Neural Networks has shown great promise in various areas of image-based classification and enhancement but is often unsuitable for predictive modeling involving non-image based features or features without spatial correlations. We present a novel approach for representation of high dimensional feature vector in a compact image form, termed REFINED (REpresentation of Features as Images with NEighborhood Dependencies), that is conducible for convolutional neural network based deep learning. We consider the correlations between features to generate a compact repre…
Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Amsterdam（阿姆斯特丹大学）；Stanford University（斯坦福大学）
The use of Convolutional Neural Networks (CNNs) is widespread in Deep Learning due to a range of desirable model properties which result in an efficient and effective machine learning framework. However, performant CNN architectures must be tailored to specific tasks in order to incorporate considerations such as the input length, resolution, and dimentionality. In this work, we overcome the need for problem-specific CNN architectures with our Continuous Convolutional Neural Network (CCNN): a single CNN architecture equipped with continuous convolutional kernels that can be used for tasks on d…
The University of Hong Kong（香港大学）；McGill University（麦吉尔大学）
Missing data is an inevitable and ubiquitous problem for traffic data collection in intelligent transportation systems. Despite extensive research regarding traffic data imputation, there still exist two limitations to be addressed: first, existing approaches fail to capture the complex spatiotemporal dependencies in traffic data, especially the dynamic spatial dependencies evolving with time; second, prior studies mainly focus on randomly missing patterns while other more complex missing scenarios are less discussed. To fill these research gaps, we propose a novel deep learning framework call…
Dynamic Frequency Domain Graph Convolutional Network for Traffic Forecasting
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Academy of Cyber（网络研究院）
Singapore University of Technology and Design（新加坡科技设计大学）；University of Moratuwa（莫拉图瓦大学）；University of Melbourne（墨尔本大学）；Keysight Technologies（是德科技）；National University of Singapore（新加坡国立大学）
Accurately estimating the remaining useful life (RUL) of industrial machinery is beneficial in many real-world applications. Estimation techniques have mainly utilized linear models or neural network based approaches with a focus on short term time dependencies. This paper, introduces a system model that incorporates temporal convolutions with both long term and short term time dependencies. The proposed network learns salient features and complex temporal variations in sensor values, and predicts the RUL. A data augmentation method is used for increased accuracy. The proposed method is compar…
University at Buffalo, The State University of New York (SUNY)（纽约州立大学布法罗分校）；Drexel University（德雷塞尔大学）；Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）
Recent advances in neuroscience and in the technology of functional magnetic resonance imaging (fMRI) and electro-encephalography (EEG) have propelled a growing interest in brain-network clustering via time-series analysis. Notwithstanding, most of the brain-network clustering methods revolve around state clustering and/or node clustering (a.k.a. community detection or topology inference) within states. This work answers first the need of capturing non-linear nodal dependencies by bringing forth a novel feature-extraction mechanism via kernel autoregressive-moving-average modeling. The extract…
A Lightweight CNN Integrated Compact Convolutional Transformer for Multi-Scale Feature Learning and reducing computational complexity for breast cancer mammography image detection and classification
