Functional magnetic resonance imaging (fMRI) is a notoriously noisy measurement of brain activity because of the large variations between individuals, signals marred by environmental differences during collection, and spatiotemporal averaging required by the measurement resolution. In addition, the data is extremely high dimensional, with the space of the activity typically having much lower intrinsic dimension. In order to understand the connection between stimuli of interest and brain activity, and analyze differences and commonalities between subjects, it becomes important to learn a meaningful embedding of the data that denoises, and reveals its intrinsic structure. Specifically, we assume that while noise varies significantly between individuals, true responses to stimuli will share common, low-dimensional features between subjects which are jointly discoverable. Similar approaches have been exploited previously but they have mainly used linear methods such as PCA and shared response modeling (SRM). In contrast, we propose a neural network called MRMD-AE (manifold-regularized multiple decoder, autoencoder), that learns a common embedding from multiple subjects in an experiment while retaining the ability to decode to individual raw fMRI signals. We show that our learned common space represents an extensible manifold (where new points not seen during training can be mapped), improves the classification accuracy of stimulus features of unseen timepoints, as well as improves cross-subject translation of fMRI signals. We believe this framework can be used for many downstream applications such as guided brain-computer interface (BCI) training in the future.
KU Leuven（鲁汶大学）；Leuven Brain Institute（鲁汶脑科学研究所）；Maastricht UMC+（马斯特里赫特大学医学中心+）；University Hospitals Leuven（鲁汶大学医院）；Delft University of Technology（代尔夫特理工大学）
EEG-correlated fMRI analysis is widely used to detect regional blood oxygen level dependent fluctuations that are significantly synchronized to interictal epileptic discharges, which can provide evidence for localizing the ictal onset zone. However, such an asymmetrical, mass-univariate approach cannot capture the inherent, higher order structure in the EEG data, nor multivariate relations in the fMRI data, and it is nontrivial to accurately handle varying neurovascular coupling over patients and brain regions. We aim to overcome these drawbacks in a data-driven manner by means of a novel stru…
Princeton University（普林斯顿大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Toronto（多伦多大学）
Several research groups have shown how to correlate fMRI responses to the meanings of presented stimuli. This paper presents new methods for doing so when only a natural language annotation is available as the description of the stimulus. We study fMRI data gathered from subjects watching an episode of BBCs Sherlock [1], and learn bidirectional mappings between fMRI responses and natural language representations. We show how to leverage data from multiple subjects watching the same movie to improve the accuracy of the mappings, allowing us to succeed at a scene classification task with 72% acc…
The objective of this paper is to provide a temporal dynamic model for resting state functional Magnetic Resonance Imaging (fMRI) trajectory to predict future brain images based on the given sequence. To this end, we came up with the model that takes advantage of representation learning and Neural Ordinary Differential Equation (Neural ODE) to compress the fMRI image data into latent representation and learn to predict the trajectory following differential equation. Latent space was analyzed by Gaussian Mixture Model. The learned fMRI trajectory embedding can be used to explain the variance of…
Brain networks in fMRI are typically identified using spatial independent component analysis (ICA), yet mathematical constraints such as sparse coding and positivity both provide alternate biologically-plausible frameworks for generating brain networks. Non-negative Matrix Factorization (NMF) would suppress negative BOLD signal by enforcing positivity. Spatial sparse coding algorithms ($L1$ Regularized Learning and K-SVD) would impose local specialization and a discouragement of multitasking, where the total observed activity in a single voxel originates from a restricted number of possible br…
Cornell University（康奈尔大学）；Weill Cornell Medical College（威尔康奈尔医学院）
The increasing popularity of naturalistic paradigms in fMRI (such as movie watching) demands novel strategies for multi-subject data analysis, such as use of neural encoding models. In the present study, we propose a shared convolutional neural encoding method that accounts for individual-level differences. Our method leverages multi-subject data to improve the prediction of subject-specific responses evoked by visual or auditory stimuli. We showcase our approach on high-resolution 7T fMRI data from the Human Connectome Project movie-watching protocol and demonstrate significant improvement ov…
University of Toronto（多伦多大学）；Centre for Addiction and Mental Health（成瘾与心理健康中心）
Multi-subject fMRI studies are challenging due to the high variability of both brain anatomy and functional brain topographies across participants. An effective way of aggregating multi-subject fMRI data is to extract a shared representation that filters out unwanted variability among subjects. Some recent work has implemented probabilistic models to extract a shared representation in task fMRI. In the present work, we improve upon these models by incorporating temporal information in the common latent structures. We introduce a new model, Shared Gaussian Process Factor Analysis (S-GPFA), that…
There is a growing interest in joint multi-subject fMRI analysis. The challenge of such analysis comes from inherent anatomical and functional variability across subjects. One approach to resolving this is a shared response factor model. This assumes a shared and time synchronized stimulus across subjects. Such a model can often identify shared information, but it may not be able to pinpoint with high resolution the spatial location of this information. In this work, we examine a searchlight based shared response model to identify shared information in small contiguous regions (searchlights) a…
The shared response model provides a simple but effective framework to analyse fMRI data of subjects exposed to naturalistic stimuli. However when the number of subjects or runs is large, fitting the model requires a large amount of memory and computational power, which limits its use in practice. In this work, we introduce the FastSRM algorithm that relies on an intermediate atlas-based representation. It provides considerable speed-up in time and memory usage, hence it allows easy and fast large-scale analysis of naturalistic-stimulus fMRI data. Using four different datasets, we show that ou…
University of Alberta（阿尔伯塔大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））；Purdue University Fort Wayne（普渡大学韦恩堡分校）
Multi-voxel pattern analysis (MVPA) learns predictive models from task-based functional magnetic resonance imaging (fMRI) data, for distinguishing when subjects are performing different cognitive tasks -- e.g., watching movies or making decisions. MVPA works best with a well-designed feature set and an adequate sample size. However, most fMRI datasets are noisy, high-dimensional, expensive to collect, and with small sample sizes. Further, training a robust, generalized predictive model that can analyze homogeneous cognitive tasks provided by multi-site fMRI datasets has additional challenges. …
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Multi-subject fMRI data analysis is an interesting and challenging problem in human brain decoding studies. The inherent anatomical and functional variability across subjects make it necessary to do both anatomical and functional alignment before classification analysis. Besides, when it comes to big data, time complexity becomes a problem that cannot be ignored. This paper proposes Gradient Hyperalignment (Gradient-HA) as a gradient-based functional alignment method that is suitable for multi-subject fMRI datasets with large amounts of samples and voxels. The advantage of Gradient-HA is that …
Columbia University（哥伦比亚大学）；Stanford University（斯坦福大学）
We propose a hierarchical Bayesian recurrent state space model for modeling switching network connectivity in resting state fMRI data. Our model allows us to uncover shared network patterns across disease conditions. We evaluate our method on the ADNI2 dataset by inferring latent state patterns corresponding to altered neural circuits in individuals with Mild Cognitive Impairment (MCI). In addition to states shared across healthy and individuals with MCI, we discover latent states that are predominantly observed in individuals with MCI. Our model outperforms current state of the art deep learn…
As a technology to read brain states from measurable brain activities, brain decoding are widely applied in industries and medical sciences. In spite of high demands in these applications for a universal decoder that can be applied to all individuals simultaneously, large variation in brain activities across individuals has limited the scope of many studies to the development of individual-specific decoders. In this study, we used deep neural network (DNN), a nonlinear hierarchical model, to construct a subject-transfer decoder. Our decoder is the first successful DNN-based subject-transfer de…
University of Regensburg（雷根斯堡大学）；Universidade de Aveiro（阿威罗大学）
Independent component analysis (ICA), as a data driven method, has shown to be a powerful tool for functional magnetic resonance imaging (fMRI) data analysis. One drawback of this multivariate approach is, that it is not compatible to the analysis of group data in general. Therefore various techniques have been proposed in order to overcome this limitation of ICA. In this paper a novel ICA-based work-flow for extracting resting state networks from fMRI group studies is proposed. An empirical mode decomposition (EMD) is used to generate reference signals in a data driven manner, which can be in…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
In order to decode the human brain, Multivariate Pattern (MVP) classification generates cognitive models by using functional Magnetic Resonance Imaging (fMRI) datasets. As a standard pipeline in the MVP analysis, brain patterns in multi-subject fMRI dataset must be mapped to a shared space and then a classification model is generated by employing the mapped patterns. However, the MVP models may not provide stable performance on a new fMRI dataset because the standard pipeline uses disjoint steps for generating these models. Indeed, each step in the pipeline includes an objective function with …
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Alberta（阿尔伯塔大学）；Purdue University Fort Wayne（普渡大学韦恩堡分校）；Manchester Metropolitan University（曼彻斯特城市大学）
Hyperalignment has been widely employed in Multivariate Pattern (MVP) analysis to discover the cognitive states in the human brains based on multi-subject functional Magnetic Resonance Imaging (fMRI) datasets. Most of the existing HA methods utilized unsupervised approaches, where they only maximized the correlation between the voxels with the same position in the time series. However, these unsupervised solutions may not be optimum for handling the functional alignment in the supervised MVP problems. This paper proposes a Supervised Hyperalignment (SHA) method to ensure better functional alig…
INESC-ID（里斯本系统与计算机工程研究所）；Instituto Superior Técnico（高等技术学院）
Topographical structures represent connections between entities and provide a comprehensive design of complex systems. Currently these structures are used to discover correlates of neuronal and haemodynamical activity. In this work, we incorporate them with neural processing techniques to perform regression, using electrophysiological activity to retrieve haemodynamics. To this end, we use Fourier features, attention mechanisms, shared space between modalities and incorporation of style in the latent representation. By combining these techniques, we propose several models that significantly ou…
Large Language Model-based FMRI Encoding of Language Functions for Subjects with Neurocognitive Disorder
The Chinese University of Hong Kong（香港中文大学）；Stanley Ho Big Data Decision Analytics Research Centre, The Chinese University of Hong Kong（香港中文大学何文汇大数据决策分析研究中心）
Automatic identification and categorization of Alzheimer's patients and the ability to distinguish between different levels of this disease can be very helpful to the research community in this field, since other non-automatic approaches are very time-consuming and are highly dependent on experts' experience. Herein, we propose the utility of cerebral instantaneous phase and envelope information in order to discriminate between Alzheimer's patients, MCI subjects and healthy normal individuals from functional magnetic resonance imaging (fMRI) data. To this end, after performing the region-of-in…
MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
Stability AI；Medical AI Research Center (MedARC)（医学人工智能研究中心（MedARC））；Princeton Neuroscience Institute（普林斯顿神经科学研究所）；University of Minnesota（明尼苏达大学）；The University of Sydney（悉尼大学）；University of Waterloo（滑铁卢大学）
Univ Rennes（雷恩大学）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；University Hospital of Rennes（雷恩大学医院）
Simultaneous EEG/fMRI acquisition allows to measure brain activity at high spatial-temporal resolution. The localisation of EEG sources depends on several parameters including the position of the electrodes on the scalp. The position of the MR electrodes during its acquisitions is obtained with the use of the UTE sequence allowing their visualisation. The retrieval of the electrodes consists in obtaining the volume where the electrodes are located by applying a sphere detection algorithm. We detect around 90% of electrodes for each subject, and our UTE-based electrode detection showed an avera…
EEG-to-fMRI synthesis of task-evoked and spontaneous brain activity: addressing issues of statistical significance and generalizability
Institute for Systems and Robotics – Lisboa and Department of Bioengineering, Instituto Superior Técnico, Universidade de Lisboa, Portugal（里斯本系统与机器人研究所及生物工程系，里斯本技术学院，里斯本大学，葡萄牙）；Electrical and Computer Engineering Department, Carnegie Mellon University（电气与计算机工程系，卡内基梅隆大学）
Johns Hopkins University（约翰斯·霍普金斯大学）；Kennedy Krieger Institute（肯尼迪·克里格研究所）；Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）
We propose a novel matrix autoencoder to map functional connectomes from resting state fMRI (rs-fMRI) to structural connectomes from Diffusion Tensor Imaging (DTI), as guided by subject-level phenotypic measures. Our specialized autoencoder infers a low dimensional manifold embedding for the rs-fMRI correlation matrices that mimics a canonical outer-product decomposition. The embedding is simultaneously used to reconstruct DTI tractography matrices via a second manifold alignment decoder and to predict inter-subject phenotypic variability via an artificial neural network. We validate our frame…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Alberta（阿尔伯塔大学）；Purdue University Fort Wayne（普渡大学韦恩堡分校）
Similarity analysis is one of the crucial steps in most fMRI studies. Representational Similarity Analysis (RSA) can measure similarities of neural signatures generated by different cognitive states. This paper develops Deep Representational Similarity Learning (DRSL), a deep extension of RSA that is appropriate for analyzing similarities between various cognitive tasks in fMRI datasets with a large number of subjects, and high-dimensionality -- such as whole-brain images. Unlike the previous methods, DRSL is not limited by a linear transformation or a restricted fixed nonlinear kernel functio…
Interpretable Information-Decomposed Brain Graph Learning for fMRI-based Disease Diagnosis
School of Mathematics and Statistics, Shandong University（山东大学数学与统计学院）；Data Science Institute, Shandong University（山东大学数据科学研究院）；Academy of Mathematics and Systems Science, University of Chinese Academy of Sciences（中国科学院大学数学与系统科学研究院）
Monash University Malaysia（莫纳什大学马来西亚分校）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University College London（伦敦大学学院）；University of Texas at Dallas（德克萨斯大学达拉斯分校）
We consider the challenges in extracting stimulus-related neural dynamics from other intrinsic processes and noise in naturalistic functional magnetic resonance imaging (fMRI). Most studies rely on inter-subject correlations (ISC) of low-level regional activity and neglect varying responses in individuals. We propose a novel, data-driven approach based on low-rank plus sparse (L+S) decomposition to isolate stimulus-driven dynamic changes in brain functional connectivity (FC) from the background noise, by exploiting shared network structure among subjects receiving the same naturalistic stimuli…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Representational Similarity Analysis (RSA) aims to explore similarities between neural activities of different stimuli. Classical RSA techniques employ the inverse of the covariance matrix to explore a linear model between the neural activities and task events. However, calculating the inverse of a large-scale covariance matrix is time-consuming and can reduce the stability and robustness of the final analysis. Notably, it becomes severe when the number of samples is too large. For facing this shortcoming, this paper proposes a novel RSA method called gradient-based RSA (GRSA). Moreover, the p…
It has become increasingly popular to study the brain as a network due to the realization that functionality cannot be explained exclusively by independent activation of specialized regions. Instead, across a large spectrum of behaviors, function arises due to the dynamic interactions between brain regions. The existing literature on functional brain networks focuses mainly on a battery of network properties characterizing the "resting state" using for example the modularity, clustering, or path length among regions. In contrast, we seek to uncover subgraphs of functional connectivity that pre…
Mind-to-Image: Projecting Visual Mental Imagination of the Brain from fMRI
Obvious Research（奥布维研究机构）；GHU Paris Psychiatrie et Neurosciences（巴黎精神病学与神经科学大学医院集团）；Université Paris Cité（巴黎西岱大学）；Centre Imagerie du Nord（北方影像中心）；Raidium（雷迪姆公司）；FrontLab, Sorbonne Université（索邦大学前沿实验室）；Institut du Cerveau（大脑研究所）；Centre de NeuroImagerie de Recherche, Sorbonne Université（索邦大学神经影像研究中心）；Institut des Systèmes Intelligents et de la Robotique, Sorbonne Université（索邦大学智能系统与机器人研究所）
a Division of Medical Informatics, Department of Biomedical Engineering b Division of Statistics \& Machine Learning, Department of Computer；Information Science c Center for Medical Image Science；Visualization (CMIV) \ \" o ping University, Sweden
Classifying subjects as healthy or diseased using neuroimaging data has gained a lot of attention during the last 10 years. Here we apply deep learning to derivatives from resting state fMRI data, and investigate how different 3D augmentation techniques affect the test accuracy. Specifically, we use resting state derivatives from 1,112 subjects in ABIDE preprocessed to train a 3D convolutional neural network (CNN) to perform the classification. Our results show that augmentation only provide minor improvements to the test accuracy.
Studies using real-time functional magnetic resonance imaging (rt-fMRI) have recently incorporated the decoding approach, allowing for fMRI to be used as a tool for manipulation of fine-grained neural activity. Because of the tremendous potential for clinical applications, certain questions regarding decoded neurofeedback (DecNef) must be addressed. Neurofeedback effects can last for months, but the short- to mid-term dynamics are not known. Specifically, can the same subjects learn to induce neural patterns in two opposite directions in different sessions? This leads to a further question, wh…
Dynamic functional connectivity (FC) has in recent years become a topic of interest in the neuroimaging community. Several models and methods exist for both functional magnetic resonance imaging (fMRI) and electroencephalography (EEG), and the results point towards the conclusion that FC exhibits dynamic changes. The existing approaches modeling dynamic connectivity have primarily been based on time-windowing the data and k-means clustering. We propose a non-parametric generative model for dynamic FC in fMRI that does not rely on specifying window lengths and number of dynamic states. Rooted i…
ETH Zurich（苏黎世联邦理工学院）；Yale University（耶鲁大学）；Mila（米拉研究所）
Functional magnetic resonance imaging (fMRI) is a crucial technology for gaining insights into cognitive processes in humans. Data amassed from fMRI measurements result in volumetric data sets that vary over time. However, analysing such data presents a challenge due to the large degree of noise and person-to-person variation in how information is represented in the brain. To address this challenge, we present a novel topological approach that encodes each time point in an fMRI data set as a persistence diagram of topological features, i.e. high-dimensional voids present in the data. This repr…
Dynamic Causal Modelling (DCM) is the predominant method for inferring effective connectivity from neuroimaging data. In the 15 years since its introduction, the neural models and statistical routines in DCM have developed in parallel, driven by the needs of researchers in cognitive and clinical neuroscience. In this tutorial, we step through an exemplar fMRI analysis in detail, reviewing the current implementation of DCM and demonstrating recent developments in group-level connectivity analysis. In the first part of the tutorial (current paper), we focus on issues specific to DCM for fMRI, un…
Much recent attention has been paid to quantifying anatomic and functional neuroimaging on the individual subject level. For optimal individual subject characterization, specific acquisition and analysis features need to be identified that maximize inter-individual variability while concomitantly minimizing intra-subject variability. Here we develop a non-parametric statistical metric that quantifies the degree to which a parameter set allows this individual subject differentiation. We apply this metric to analyzing publicly available test-retest resting-state fMRI (rs-fMRI) data sets. We find…
While objects from different categories can be reliably decoded from fMRI brain response patterns, it has proved more difficult to distinguish visually similar inputs, such as different instances of the same category. Here, we apply a recently developed deep learning system to the reconstruction of face images from human fMRI patterns. We trained a variational auto-encoder (VAE) neural network using a GAN (Generative Adversarial Network) unsupervised training procedure over a large dataset of celebrity faces. The auto-encoder latent space provides a meaningful, topologically organized 1024-dim…
It has been well established that Functional Connectomes (FCs), as estimated from functional MRI (fMRI) data, have an individual fingerprint that can be used to identify an individual from a population (subject-identification). Although identification rate is high when using resting-state FCs, other tasks show moderate to low values. Furthermore, identification rate is task-dependent, and is low when distinct cognitive states, as captured by different fMRI tasks, are compared. Here we propose an embedding framework, GEFF (Graph Embedding for Functional Fingerprinting), based on group-level dec…
Functional Brain Network Identification in Opioid Use Disorder Using Machine Learning Analysis of Resting-State fMRI BOLD Signals
Old Dominion University（奥多明尼昂大学）；Institute of Drug and Alcohol Studies（药物与酒精研究所）；Virginia Commonwealth University（弗吉尼亚联邦大学）；C. Kenneth and Dianne Wright Center for Clinical and Translational Research（C.肯尼斯和黛安·赖特临床与转化研究中心）；Data Science Institute（数据科学研究所）
Estimated connectomes by the means of neuroimaging techniques have enriched our knowledge of the organizational properties of the brain leading to the development of network-based clinical diagnostics. Unfortunately, to date, many of those network-based clinical diagnostics tools, based on the mere description of isolated instances of observed connectomes are noisy estimates of the true connectivity network. Modeling brain connectivity networks is therefore important to better explain the functional organization of the brain and allow inference of specific brain properties. In this report, we …
Pre-surgical language mapping with functional magnetic resonance imaging (fMRI) is routinely conducted to assist the neurosurgeon in preventing damage to brain regions responsible for language. Functional differences exist between the monolingual versus the bilingual brain, whereas clinical fMRI tasks are typically conducted in a single language. The presence of secondary language processing mechanisms is a potential source of error in the inferred language map. From fMRI data of healthy bilingual and monolingual subjects we obtain language maps as functional networks. Our results show a sub-n…
Brain regions are often topographically connected: nearby locations within one brain area connect with nearby locations in another area. Mapping these connection topographies, or 'connectopies' in short, is crucial for understanding how information is processed in the brain. Here, we propose principled, fully data-driven methods for mapping connectopies using functional magnetic resonance imaging (fMRI) data acquired at rest by combining spectral embedding of voxel-wise connectivity 'fingerprints' with a novel approach to spatial statistical inference. We applied the approach in human primary …
The University of Tokyo（东京大学）；Princeton University（普林斯顿大学）；Facebook Reality Labs（Facebook现实实验室）；Columbia University（哥伦比亚大学）；Intel Corporation（英特尔公司）
With the wide adoption of functional magnetic resonance imaging (fMRI) by cognitive neuroscience researchers, large volumes of brain imaging data have been accumulated in recent years. Aggregating these data to derive scientific insights often faces the challenge that fMRI data are high-dimensional, heterogeneous across people, and noisy. These challenges demand the development of computational tools that are tailored both for the neuroscience questions and for the properties of the data. We review a few recently developed algorithms in various domains of fMRI research: fMRI in naturalistic ta…
