Towards a predictive spatio-temporal representation of brain data
University of Cambridge（剑桥大学）；University of Rome “Tor Vergata”（罗马大学“Tor Vergata”）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）
Tiago Azevedo, Luca Passamonti, Pietro Liò, Nicola Toschi
备注：
To appear in the Workshop on AI for Affordable Healthcare (AI4AH) at ICLR 2020. 8 pages, 2 figures
The characterisation of the brain as a "connectome", in which the connections are represented by correlational values across timeseries and as summary measures derived from graph theory analyses, has been very popular in the last years. However, although this representation has advanced our understanding of the brain function, it may represent an oversimplified model. This is because the typical fMRI datasets are constituted by complex and highly heterogeneous timeseries that vary across space (i.e., location of brain regions). We compare various modelling techniques from deep learning and geometric deep learning to pave the way for future research in effectively leveraging the rich spatial and temporal domains of typical fMRI datasets, as well as of other similar datasets. As a proof-of-concept, we compare our approaches in the homogeneous and publicly available Human Connectome Project (HCP) dataset on a supervised binary classification task. We hope that our methodological advances relative to previous "connectomic" measures can ultimately be clinically and computationally relevant by leading to a more nuanced understanding of the brain dynamics in health and disease. Such understanding of the brain can fundamentally reduce the constant specialised clinical expertise in order to accurately understand brain variability.
Comprehending the interplay between spatial and temporal characteristics of neural dynamics can contribute to our understanding of information processing in the human brain. Graph neural networks (GNNs) provide a new possibility to interpret graph structured signals like those observed in complex brain networks. In our study we compare different spatio-temporal GNN architectures and study their ability to model neural activity distributions obtained in functional MRI (fMRI) studies. We evaluate the performance of the GNN models on a variety of scenarios in MRI studies and also compare it to a …
University College London（伦敦大学学院）；University of Oxford（牛津大学）；The Alan Turing Institute（艾伦·图灵研究所）
As a relatively new field, network neuroscience has tended to focus on aggregate behaviours of the brain averaged over many successive experiments or over long recordings in order to construct robust brain models. These models are limited in their ability to explain dynamic state changes in the brain which occurs spontaneously as a result of normal brain function. Hidden Markov Models (HMMs) trained on neuroimaging time series data have since arisen as a method to produce dynamical models that are easy to train but can be difficult to fully parametrise or analyse. We propose an interpretation …
BrainCast: A Spatio-Temporal Forecasting Model for Whole-Brain fMRI Time Series Prediction
Institute of Advanced Technology, University of Science and Technology of China（科学技术大学先进技术研究院）；MoE Key Laboratory of Brain-Inspired Intelligence Perception and Cognition, University of Science and Technology of China（科学技术大学脑启发智能感知与认知教育部重点实验室）；School of Mathematics and Computer Sciences, Nanchang University（南昌大学数学与计算机科学学院）；School of Data Science, University of Virginia（弗吉尼亚大学数据科学学院）；Department of Biomedical Engineering, Tulane University（路易斯安那大学医学工程系）
School of Software, BNRist, Research Center for Big Data, Tsinghua University（清华大学软件学院、北京智源人工智能研究院、大数据研究中心）
This paper explores a new research problem of unsupervised transfer learning across multiple spatiotemporal prediction tasks. Unlike most existing transfer learning methods that focus on fixing the discrepancy between supervised tasks, we study how to transfer knowledge from a zoo of unsupervisedly learned models towards another predictive network. Our motivation is that models from different sources are expected to understand the complex spatiotemporal dynamics from different perspectives, thereby effectively supplementing the new task, even if the task has sufficient training samples. Techni…
The Blood-Oxygen-Level-Dependent (BOLD) signal of resting-state fMRI (rs-fMRI) records the temporal dynamics of intrinsic functional networks in the brain. However, existing deep learning methods applied to rs-fMRI either neglect the functional dependency between different brain regions in a network or discard the information in the temporal dynamics of brain activity. To overcome those shortcomings, we propose to formulate functional connectivity networks within the context of spatio-temporal graphs. We train a spatio-temporal graph convolutional network (ST-GCN) on short sub-sequences of the…
We present PredRNN++, an improved recurrent network for video predictive learning. In pursuit of a greater spatiotemporal modeling capability, our approach increases the transition depth between adjacent states by leveraging a novel recurrent unit, which is named Causal LSTM for re-organizing the spatial and temporal memories in a cascaded mechanism. However, there is still a dilemma in video predictive learning: increasingly deep-in-time models have been designed for capturing complex variations, while introducing more difficulties in the gradient back-propagation. To alleviate this undesirab…
Functional Magnetic Resonance Imaging (fMRI) is a primary modality for studying brain activity. Modeling spatial dependence of imaging data at different scales is one of the main challenges of contemporary neuroimaging, and it could allow for accurate testing for significance in neural activity. The high dimensionality of this type of data (on the order of hundreds of thousands of voxels) poses serious modeling challenges and considerable computational constraints. For the sake of feasibility, standard models typically reduce dimensionality by modeling covariance among regions of interest (ROI…
Spatio-temporal data is intrinsically high dimensional, so unsupervised modeling is only feasible if we can exploit structure in the process. When the dynamics are local in both space and time, this structure can be exploited by splitting the global field into many lower-dimensional "light cones". We review light cone decompositions for predictive state reconstruction, introducing three simple light cone algorithms. These methods allow for tractable inference of spatio-temporal data, such as full-frame video. The algorithms make few assumptions on the underlying process yet have good predictiv…
MEG and EEG are noninvasive functional neuroimaging techniques that provide recordings of brain activity with high temporal resolution, and thus provide a unique window to study fast time-scale neural dynamics in humans. However, the accuracy of brain activity estimates resulting from these data is limited mainly because 1) the number of sensors is much smaller than the number of sources, and 2) the low sensitivity of the recording device to deep or radially oriented sources. These factors limit the number of sources that can be recovered and bias estimates to superficial cortical areas, resul…
Natural spatiotemporal processes can be highly non-stationary in many ways, e.g. the low-level non-stationarity such as spatial correlations or temporal dependencies of local pixel values; and the high-level variations such as the accumulation, deformation or dissipation of radar echoes in precipitation forecasting. From Cramer's Decomposition, any non-stationary process can be decomposed into deterministic, time-variant polynomials, plus a zero-mean stochastic term. By applying differencing operations appropriately, we may turn time-variant polynomials into a constant, making the deterministi…
Deep learning has achieved incredible success over the past years, especially in various challenging predictive spatio-temporal analytics (PSTA) tasks, such as disease prediction, climate forecast, and traffic prediction, where intrinsic dependency relationships among data exist and generally manifest at multiple spatio-temporal scales. However, given a specific PSTA task and the corresponding dataset, how to appropriately determine the desired configuration of a deep learning model, theoretically analyze the model's learning behavior, and quantitatively characterize the model's learning capac…
University College London（伦敦大学学院）；Inria（法国国家信息与自动化研究所）；Université Côte d'Azur（蔚蓝海岸大学）
Here we present DIVE: Data-driven Inference of Vertexwise Evolution. DIVE is an image-based disease progression model with single-vertex resolution, designed to reconstruct long-term patterns of brain pathology from short-term longitudinal data sets. DIVE clusters vertex-wise biomarker measurements on the cortical surface that have similar temporal dynamics across a patient population, and concurrently estimates an average trajectory of vertex measurements in each cluster. DIVE uniquely outputs a parcellation of the cortex into areas with common progression patterns, leading to a new signature…
School of Information Technology, University of Cincinnati（辛辛那提大学信息技术学院）；School of Computing and Informatics, University of Louisiana at Lafayette（路易斯安那大学拉法叶分校计算与信息学院）
Spatiotemporal sequence prediction is an important problem in deep learning. We study next-frame(s) video prediction using a deep-learning-based predictive coding framework that uses convolutional, long short-term memory (convLSTM) modules. We introduce a novel reduced-gate convolutional LSTM(rgcLSTM) architecture that requires a significantly lower parameter budget than a comparable convLSTM. By using a single multi-function gate, our reduced-gate model achieves equal or better next-frame(s) prediction accuracy than the original convolutional LSTM while using a smaller parameter budget, there…
MEG/EEG are non-invasive imaging techniques that record brain activity with high temporal resolution. However, estimation of brain source currents from surface recordings requires solving an ill-posed inverse problem. Converging lines of evidence in neuroscience, from neuronal network models to resting-state imaging and neurophysiology, suggest that cortical activation is a distributed spatiotemporal dynamic process, supported by both local and long-distance neuroanatomic connections. Because spatiotemporal dynamics of this kind are central to brain physiology, inverse solutions could be impro…
Compared with artificial neural networks (ANNs), spiking neural networks (SNNs) are promising to explore the brain-like behaviors since the spikes could encode more spatio-temporal information. Although pre-training from ANN or direct training based on backpropagation (BP) makes the supervised training of SNNs possible, these methods only exploit the networks' spatial domain information which leads to the performance bottleneck and requires many complicated training skills. Another fundamental issue is that the spike activity is naturally non-differentiable which causes great difficulties in t…
This article concerns the predictive modeling for spatio-temporal data as well as model interpretation using data information in space and time. We develop a novel approach based on supervised dimension reduction for such data in order to capture nonlinear mean structures without requiring a prespecified parametric model. In addition to prediction as a common interest, this approach emphasizes the exploration of geometric information from the data. The method of Pairwise Directions Estimation (PDE; Lue, 2019) is implemented in our approach as a data-driven function searching for spatial patter…
We present a new, non-parametric forecasting method for data where continuous values are observed discretely in space and time. Our method, "light-cone reconstruction of states" (LICORS), uses physical principles to identify predictive states which are local properties of the system, both in space and time. LICORS discovers the number of predictive states and their predictive distributions automatically, and consistently, under mild assumptions on the data source. We provide an algorithm to implement our method, along with a cross-validation scheme to pick control settings. Simulations show th…
We introduce a probabilistic generative model for disentangling spatio-temporal disease trajectories from series of high-dimensional brain images. The model is based on spatio-temporal matrix factorization, where inference on the sources is constrained by anatomically plausible statistical priors. To model realistic trajectories, the temporal sources are defined as monotonic and time-reparametrized Gaussian Processes. To account for the non-stationarity of brain images, we model the spatial sources as sparse codes convolved at multiple scales. The method was tested on synthetic data favourably…
The traditional methods for detecting autism spectrum disorder (ASD) are expensive, subjective, and time-consuming, often taking years for a diagnosis, with many children growing well into adolescence and even adulthood before finally confirming the disorder. Recently, graph-based learning techniques have demonstrated impressive results on resting-state functional magnetic resonance imaging (rs-fMRI) data from the Autism Brain Imaging Data Exchange (ABIDE). We introduce IMAGIN, a multI-granular, Multi-Atlas spatio-temporal attention Graph Isomorphism Network, which we use to learn graph repres…
Directorate of Technology and Innovation（技术与创新局）；Chamber of Deputies（众议院）
Deep predictive coding networks are neuroscience-inspired unsupervised learning models that learn to predict future sensory states. We build upon the PredNet implementation by Lotter, Kreiman, and Cox (2016) to investigate if predictive coding representations are useful to predict brain activity in the visual cortex. We use representational similarity analysis (RSA) to compare PredNet representations to functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data from the Algonauts Project. In contrast to previous findings in the literature (Khaligh-Razavi &Kriegeskorte, …
Functional connectivity (FC) between regions of the brain can be assessed by the degree of temporal correlation measured with functional neuroimaging modalities. Based on the fact that these connectivities build a network, graph-based approaches for analyzing the brain connectome have provided insights into the functions of the human brain. The development of graph neural networks (GNNs) capable of learning representation from graph structured data has led to increased interest in learning the graph representation of the brain connectome. Although recent attempts to apply GNN to the FC network…
Targeted stimulation of the brain has the potential to treat mental illnesses. We propose an approach to help design the stimulation protocol by identifying electrical dynamics across many brain regions that relate to illness states. We model multi-region electrical activity as a superposition of activity from latent networks, where the weights on the latent networks relate to an outcome of interest. In order to improve on drawbacks of latent factor modeling in this context, we focus on supervised autoencoders (SAEs), which can improve predictive performance while maintaining a generative mode…
Amsterdam UMC（阿姆斯特丹大学医学中心）；University of Amsterdam（阿姆斯特丹大学）
The characterisation of the brain as a functional network in which the connections between brain regions are represented by correlation values across time series has been very popular in the last years. Although this representation has advanced our understanding of brain function, it represents a simplified model of brain connectivity that has a complex dynamic spatio-temporal nature. Oversimplification of the data may hinder the merits of applying advanced non-linear feature extraction algorithms. To this end, we propose a dynamic adaptive spatio-temporal graph convolution (DAST-GCN) model to…
Otto von Guericke University Magdeburg（奥托·冯·格里克马格德堡大学）；Center for Behavioral Brain Sciences（行为脑科学中心）；German Center for Neurodegenerative Disease（德国神经退行性疾病中心）；Leibniz Institute for Neurobiology（莱布尼茨神经生物学研究所）
A brain tumour is a mass or cluster of abnormal cells in the brain, which has the possibility of becoming life-threatening because of its ability to invade neighbouring tissues and also form metastases. An accurate diagnosis is essential for successful treatment planning and magnetic resonance imaging is the principal imaging modality for diagnostic of brain tumours and their extent. Deep Learning methods in computer vision applications have shown significant improvement in recent years, most of which can be credited to the fact that a sizeable amount of data is available to train models on, a…
DCL-SE: Dynamic Curriculum Learning for Spatiotemporal Encoding of Brain Imaging
University of Chinese Academy of Sciences（中国科学院大学）；Capital Medical University（首都医科大学）；Beijing Tongren Hospital（北京同仁医院）；Beijing Institute of Ophthalmology（北京眼科研究所）；Wannan Medical University（皖南医学院）
Interpretable Spatio-Temporal Embedding for Brain Structural-Effective Network with Ordinary Differential Equation
University of Texas Rio Grande Valley（德克萨斯大学里奥格兰德河谷分校）；University of Maryland（马里兰大学）；University of Pittsburgh（匹兹堡大学）；Texas A&M University - Corpus Christi（德克萨斯农工大学科珀斯克里斯蒂分校）；Emory University（埃默里大学）；Lehigh University（利哈伊大学）；University of Illinois at Chicago（芝加哥伊利诺伊大学）；University of Southern California（南加州大学）
STARFormer: A Novel Spatio-Temporal Aggregation Reorganization Transformer of FMRI for Brain Disorder Diagnosis
Laboratory of Digital Image and Intelligent Computation, College of Information Engineering, Shanghai Maritime University（数字图像与智能计算实验室，信息工程学院，上海 Maritime 大学）；Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University（中英双语研究部，香港理工大学）；Department of Neurology, Affiliated Lianyungang Hospital of Xuzhou Medical University（神经科，徐州医学院附属连云港医院）
针对 fMRI 脑疾病分类忽略时空依赖的问题，提出 STARFormer，通过 ROI 空间重组、时间特征重组和时空融合三个模块，在 ASD 和 ADHD 数据集上达到最先进性能。
Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken s…
A Deep Spatio-Temporal Architecture for Dynamic Effective Connectivity Network Analysis Based on Dynamic Causal Discovery
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；Department of Biomedical Engineering, Tulane University（杜兰大学生物医学工程系）；Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University（佐治亚州立大学、佐治亚理工学院、埃默里大学三方转化神经成像与数据科学中心（TReNDS））；Mind Research Network（心智研究网络）；Institute for Human Neuroscience, Boys Town National Research Hospital（博伊斯镇国家医院人类神经科学研究所）
Imperial College London（帝国理工学院）；Technical University of Munich（慕尼黑工业大学）
The study of functional brain connectivity (FC) is important for understanding the underlying mechanisms of many psychiatric disorders. Many recent analyses adopt graph convolutional networks, to study non-linear interactions between functionally-correlated states. However, although patterns of brain activation are known to be hierarchically organised in both space and time, many methods have failed to extract powerful spatio-temporal features. To overcome those challenges, and improve understanding of long-range functional dynamics, we translate an approach, from the domain of skeleton-based …
Brown University（布朗大学）；Université de Toulouse（图卢兹大学）
Our ability to generalize beyond training data to novel, out-of-distribution, image degradations is a hallmark of primate vision. The predictive brain, exemplified by predictive coding networks (PCNs), has become a prominent neuroscience theory of neural computation. Motivated by the recent successes of variational autoencoders (VAEs) in machine learning, we rigorously derive a correspondence between PCNs and VAEs. This motivates us to consider iterative extensions of VAEs (iVAEs) as plausible variational extensions of the PCNs. We further demonstrate that iVAEs generalize to distributional sh…
BNRist, Tsinghua University（清华大学BNRist）；AI Institute, Shanghai Jiao Tong University（上海交通大学AI研究院）
Learning predictive models for unlabeled spatiotemporal data is challenging in part because visual dynamics can be highly entangled in real scenes, making existing approaches prone to overfit partial modes of physical processes while neglecting to reason about others. We name this phenomenon spatiotemporal mode collapse and explore it for the first time in predictive learning. The key is to provide the model with a strong inductive bias to discover the compositional structures of latent modes. To this end, we propose ModeRNN, which introduces a novel method to learn structured hidden represent…
Why the Brain Consolidates: Predictive Forgetting for Optimal Generalisation
Huawei Noah’s Ark Lab（华为诺亚实验室）；AI Centre, Department of Computer Science, University College London（大学计算机科学系人工智能中心）；UCL Institute of Cognitive Neuroscience, University College London（大学认知神经科学研究所）；UCL Queen Square Institute of Neurology, University College London（大学Queen Square神经学研究所）
CINeMA: Conditional Implicit Neural Multi-Modal Atlas for a Spatio-Temporal Representation of the Perinatal Brain
School of Computation, Information and Technology, and the School of Medicine and Health（计算信息学院及医学健康学院）；Technical University of Munich（慕尼黑技术大学）；Department of Computing（计算系）；Institut de Neurosciences de la Timone, UMR 7289, CNRS, Aix-Marseille Université（神经科学研究所，UMR 7289，CNRS，艾克斯-马赛大学）；Aix-Marseille Univ, APHM, Service de Neuroradiologie Diagnostique et Interventionnelle, Hôpital de la Timone（艾克斯-马赛大学，APHM，诊断与介入神经放射科，泰米翁医院）；Aix-Marseille Univ, APHM, service de neurologie pédiatrique, Hôpital de la Timone（艾克斯-马赛大学，APHM，儿童神经科，泰米翁医院）
Technological University Dublin（都柏林理工大学）；Stanford University（斯坦福大学）；Ford Motor Company（福特汽车公司）
Brain tissue deformation resulting from head impacts is primarily caused by rotation and can lead to traumatic brain injury. To quantify brain injury risk based on measurements of kinematics on the head, finite element (FE) models and various brain injury criteria based on different factors of these kinematics have been developed, but the contribution of different kinematic factors has not been comprehensively analyzed across different types of head impacts in a data-driven manner. To better design brain injury criteria, the predictive power of rotational kinematics factors, which are differen…
Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Survey
National Innovative Institute of Defense Technology（国防科技创新研究院）；National University of Defense Technology（国防科技大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Electronic Science and Technology of China（电子科技大学）；Institute of Computing Technology, University of Chinese Academy of Sciences（中国科学院大学计算技术研究所）；College of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院）；JD Intelligent Cities Research（京东智能城市研究院）；JD iCity, JD Technology（京东科技京东智城）
University of Mons（蒙斯大学）；IMT Nord Europe（北欧国立高等矿业电信学院）；CNRS（法国国家科学研究中心）
For many years now, understanding the brain mechanism has been a great research subject in many different fields. Brain signal processing and especially electroencephalogram (EEG) has recently known a growing interest both in academia and industry. One of the main examples is the increasing number of Brain-Computer Interfaces (BCI) aiming to link brains and computers. In this paper, we present a novel framework allowing us to retrieve the attention state, i.e degree of attention given to a specific task, from EEG signals. While previous methods often consider the spatial relationship in EEG th…
