Learning in brain-computer interface control evidenced by joint decomposition of brain and behavior
Jennifer Stiso, Marie-Constance Corsi, Jean M. Vettel, Javier O. Garcia, Fabio Pasqualetti, Fabrizio De Vico Fallani, Timothy H. Lucas, Danielle S. Bassett
Motor imagery-based brain-computer interfaces (BCIs) use an individuals ability to volitionally modulate localized brain activity as a therapy for motor dysfunction or to probe causal relations between brain activity and behavior. However, many individuals cannot learn to successfully modulate their brain activity, greatly limiting the efficacy of BCI for therapy and for basic scientific inquiry. Previous research suggests that coherent activity across diverse cognitive systems is a hallmark of individuals who can successfully learn to control the BCI. However, little is known about how these distributed networks interact through time to support learning. Here, we address this gap in knowledge by constructing and applying a multimodal network approach to decipher brain-behavior relations in motor imagery-based brain-computer interface learning using MEG. Specifically, we employ a minimally constrained matrix decomposition method (non-negative matrix factorization) to simultaneously identify regularized, covarying subgraphs of functional connectivity, to assess their similarity to task performance, and to detect their time-varying expression. Individuals also displayed marked variation in the spatial properties of subgraphs such as the connectivity between the frontal lobe and the rest of the brain, and in the temporal properties of subgraphs such as the stage of learning at which they reached maximum expression. From these observations, we posit a conceptual model in which certain subgraphs support learning by modulating brain activity in regions important for sustaining attention. To test this model, we use tools that stipulate regional dynamics on a networked system (network control theory), and find that good learners display a single subgraph whose temporal expression tracked performance and whose architecture supports easy modulation of brain regions important for attention.
BCMI-Driven Motion Control Detection: EEG-Based Machine Learning and Interaction Entropy for High-Order Brain Networks
College of Information and Control Engineering, Xi'an University of Architecture and Technology（西安建筑科技大学信息与控制工程学院）；Department of Neurosurgery, General Hospital of Central Theater Command（中部战区总医院神经外科）
Recent studies have shown that multi-modeling methods can provide new insights into the analysis of brain components that are not possible when each modality is acquired separately. The joint representations of different modalities is a robust model to analyze simultaneously acquired electroencephalography and functional magnetic resonance imaging (EEG-fMRI). Advances in precision instruments have given us the ability to observe the spatiotemporal neural dynamics of the human brain through non-invasive neuroimaging techniques such as EEG & fMRI. Nonlinear fusion methods of streams can extract …
The white-matter (micro-)structural architecture of the brain promotes synchrony among neuronal populations, giving rise to richly patterned functional connections. A fundamental problem for systems neuroscience is determining the best way to relate structural and functional networks quantified by diffusion tensor imaging and resting-state functional MRI. As one of the state-of-the-art approaches for network analysis, graph convolutional networks (GCN) have been separately used to analyze functional and structural networks, but have not been applied to explore inter-network relationships. In t…
Most humans have the good fortune to live their lives embedded in richly structured social groups. Yet, it remains unclear how humans acquire knowledge about these social structures to successfully navigate social relationships. Here we address this knowledge gap with an interdisciplinary neuroimaging study drawing on recent advances in network science and statistical learning. Specifically, we collected BOLD MRI data while participants learned the community structure of both social and non-social networks, in order to examine whether the learning of these two types of networks was differentia…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Emory University（埃默里大学）；Lehigh University（理海大学）
Multimodal brain networks characterize complex connectivities among different brain regions from both structural and functional aspects and provide a new means for mental disease analysis. Recently, Graph Neural Networks (GNNs) have become a de facto model for analyzing graph-structured data. However, how to employ GNNs to extract effective representations from brain networks in multiple modalities remains rarely explored. Moreover, as brain networks provide no initial node features, how to design informative node attributes and leverage edge weights for GNNs to learn is left unsolved. To this…
This paper discusses the use of the scalp electric field to decode EEG-recorded brain processes. Instead of using bipolar measurements, the scalp electric field is described as a 3-vector field whose orthogonal components are obtained from the data through spline differentiation. The method was tested in the context of brainwave recognition with experiments involving brain representation of spoken phonemes, visual images, and mental images. The practical effect of improvements in recognition rates was assessed by estimating effect sizes and confidence intervals, the results suggesting a good p…
In this paper, the deep learning (DL) approach is applied to a joint training scheme for asynchronous motor imagery-based Brain-Computer Interface (BCI). The proposed DL approach is a cascade of one-dimensional convolutional neural networks and fully-connected neural networks (CNN-FC). The focus is mainly on three types of brain responses: non-imagery EEG (\textit{background EEG}), (\textit{pure imagery}) EEG, and EEG during the transitional period between background EEG and pure imagery (\textit{transitional imagery}). The study of transitional imagery signals should provide greater insight i…
Ludwig-Maximilians-University（慕尼黑大学）；Technical University of Munich（慕尼黑工业大学）
The reconstruction of cerebral cortex surfaces from brain MRI scans is instrumental for the analysis of brain morphology and the detection of cortical thinning in neurodegenerative diseases like Alzheimer's disease (AD). Moreover, for a fine-grained analysis of atrophy patterns, the parcellation of the cortical surfaces into individual brain regions is required. For the former task, powerful deep learning approaches, which provide highly accurate brain surfaces of tissue boundaries from input MRI scans in seconds, have recently been proposed. However, these methods do not come with the ability…
Designing brain-computer interfaces (BCIs) that can be used in conjunction with ongoing motor behavior requires an understanding of how neural activity co-opted for brain control interacts with existing neural circuits. For example, BCIs may be used to regain lost motor function after stroke. This requires that neural activity controlling unaffected limbs is dissociated from activity controlling the BCI. In this study we investigated how primary motor cortex accomplishes simultaneous BCI control and motor control in a task that explicitly required both activities to be driven from the same bra…
Facebook AI Research（Facebook AI研究院）；Université de Paris（巴黎大学）；CoML, ENS/CNRS/EHESS/INRIA/PSL University（计算语言学实验室，巴黎高等师范学院/法国国家科学研究中心/法国社会科学高等研究院/法国国家信息与自动化研究所/巴黎文理研究大学）；CRI, Département Frontières du Vivant et de l’Apprendre, IIFR, Université de Paris（认知研究所，生命与学习前沿系，法国医学研究院，巴黎大学）；École normale supérieure, PSL University, CNRS（巴黎高等师范学院，巴黎文理研究大学，法国国家科学研究中心）
Our ability to comprehend speech remains, to date, unrivaled by deep learning models. This feat could result from the brain's ability to fine-tune generic sound representations for speech-specific processes. To test this hypothesis, we compare i) five types of deep neural networks to ii) human brain responses elicited by spoken sentences and recorded in 102 Dutch subjects using functional Magnetic Resonance Imaging (fMRI). Each network was either trained on an acoustics scene classification, a speech-to-text task (based on Bengali, English, or Dutch), or not trained. The similarity between eac…
The Center for Brains, Minds and Machines（大脑、心智与机器中心）；Massachusetts Institute of Technology（麻省理工学院）
Modern high-scoring models of vision in the brain score competition do not stem from Vision Transformers. However, in this paper, we provide evidence against the unexpected trend of Vision Transformers (ViT) being not perceptually aligned with human visual representations by showing how a dual-stream Transformer, a CrossViT$~\textit{a la}$ Chen et al. (2021), under a joint rotationally-invariant and adversarial optimization procedure yields 2nd place in the aggregate Brain-Score 2022 competition(Schrimpf et al., 2020b) averaged across all visual categories, and at the time of the competition h…
JOSA: Joint surface-based registration and atlas construction of brain geometry and function
McGovern Institute for Brain Research, MIT（麦戈文脑科学研究所（麻省理工学院））；Harvard University（哈佛大学）；Computer Science and Artificial Intelligence Laboratory, MIT（计算机科学与人工智能实验室（麻省理工学院））
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；Vanderbilt University Institute of Imaging Science（范德堡大学影像科学研究所）；Vanderbilt Brain Institute（范德堡脑科学研究所）
Mild traumatic brain injury (mTBI) is a complex syndrome that affects up to 600 per 100,000 individuals, with a particular concentration among military personnel. About half of all mTBI patients experience a diverse array of chronic symptoms which persist long after the acute injury. Hence, there is an urgent need for better understanding of the white matter and gray matter pathologies associated with mTBI to map which specific brain systems are impacted and identify courses of intervention. Previous works have linked mTBI to disruptions in white matter pathways and cortical surface abnormalit…
University of California, Berkeley（加州大学伯克利分校）；University of Virginia（弗吉尼亚大学）
Determining functional brain connectivity is crucial to understanding the brain and neural differences underlying disorders such as autism. Recent studies have used Gaussian graphical models to learn brain connectivity via statistical dependencies across brain regions from neuroimaging. However, previous studies often fail to properly incorporate priors tailored to neuroscience, such as preferring shorter connections. To remedy this problem, the paper here introduces a novel, weighted-$\ell_1$, multi-task graphical model (W-SIMULE). This model elegantly incorporates a flexible prior, along wit…
This paper considers joint learning of multiple sparse Granger graphical models to discover underlying common and differential Granger causality (GC) structures across multiple time series. This can be applied to drawing group-level brain connectivity inferences from a homogeneous group of subjects or discovering network differences among groups of signals collected under heterogeneous conditions. By recognizing that the GC of a single multivariate time series can be characterized by common zeros of vector autoregressive (VAR) lag coefficients, a group sparse prior is included in joint regular…
It is widely believed that the formation of brain network structure is under the pressure of optimal trade-off between reducing wiring cost and promoting communication efficiency. However, the question of whether this trade-off exists in empirical human brain networks and, if so, how it takes effect is still not well understood. Here, we employed a multiobjective evolutionary algorithm to directly and quantitatively explore the cost-efficiency trade-off in human brain networks. Using this algorithm, we generated a population of synthetic networks with optimal but diverse cost-efficiency trade-…
Self-localization during navigation with noisy sensors in an ambiguous world is computationally challenging, yet animals and humans excel at it. In robotics, Simultaneous Location and Mapping (SLAM) algorithms solve this problem though joint sequential probabilistic inference of their own coordinates and those of external spatial landmarks. We generate the first neural solution to the SLAM problem by training recurrent LSTM networks to perform a set of hard 2D navigation tasks that include generalization to completely novel trajectories and environments. The hidden unit representations exhibit…
Introduction: Two key dimensions of the mind are understanding and responding to another's mental state (empathizing), and analysing lawful behaviour (systemizing). Methods: Two questionnaires, the Systemizing Quotient (SQ) and the Empathy Quotient (EQ), were administered to a normal control group and a group of individuals with Asperger Syndrome (AS) or High-Functioning Autism (HFA). The multivariate correlations of the joint scores were analysed using principal components analysis. Results: The principal components were well-approximated by the sums and differences of the SQ and EQ scores. T…
Neural networks with equal excitatory and inhibitory feedback show high computational performance. They operate close to a critical point characterized by the joint activation of large populations of neurons. Yet, in macaque motor cortex we observe very different dynamics with weak fluctuations on the population level. This suggests that motor cortex operates in a sub-optimal regime. Here we show the opposite: the large dispersion of correlations across neurons is a signature of a rich dynamical repertoire, hidden from macroscopic brain signals, but essential for high performance in such conce…
Sensory feedback is critical to the performance of neural prostheses that restore movement control after neurological injury. Recent advances in direct neural control of paralyzed arms present new requirements for miniaturized, low-power sensor systems. To address this challenge, we developed a fully-integrated wireless sensor-brain-machine interface (SBMI) system for communicating key somatosensory signals, fingertip forces and limb joint angles, to the brain. The system consists of a tactile force sensor, an electrogoniometer, and a neural interface. The tactile force sensor features a novel…
Emory University（埃默里大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Princeton University（普林斯顿大学）；Lehigh University（理海大学）
Interpretable brain network models for disease prediction are of great value for the advancement of neuroscience. GNNs are promising to model complicated network data, but they are prone to overfitting and suffer from poor interpretability, which prevents their usage in decision-critical scenarios like healthcare. To bridge this gap, we propose BrainNNExplainer, an interpretable GNN framework for brain network analysis. It is mainly composed of two jointly learned modules: a backbone prediction model that is specifically designed for brain networks and an explanation generator that highlights …
Center for Neurotechnology（神经技术中心）；Paul G. Allen School of Computer Science and Engineering（保罗·G·艾伦计算机科学与工程学院）；University of Washington（华盛顿大学）
The field of brain-computer interfaces is poised to advance from the traditional goal of controlling prosthetic devices using brain signals to combining neural decoding and encoding within a single neuroprosthetic device. Such a device acts as a "co-processor" for the brain, with applications ranging from inducing Hebbian plasticity for rehabilitation after brain injury to reanimating paralyzed limbs and enhancing memory. We review recent progress in simultaneous decoding and encoding for closed-loop control and plasticity induction. To address the challenge of multi-channel decoding and encod…
Center for Neurotechnology（神经技术中心）；Paul G. Allen School of Computer Science and Engineering（保罗·G·艾伦计算机科学与工程学院）；University of Washington（华盛顿大学）
Brain-computer interfaces (BCIs) use decoding algorithms to control prosthetic devices based on brain signals for restoration of lost function. Computer-brain interfaces (CBIs), on the other hand, use encoding algorithms to transform external sensory signals into neural stimulation patterns for restoring sensation or providing sensory feedback for closed-loop prosthetic control. In this article, we introduce brain co-processors, devices that combine decoding and encoding in a unified framework using artificial intelligence (AI) to supplement or augment brain function. Brain co-processors can b…
Cataloging the neuronal cell types that comprise circuitry of individual brain regions is a major goal of modern neuroscience and the BRAIN initiative. Single-cell RNA sequencing can now be used to measure the gene expression profiles of individual neurons and to categorize neurons based on their gene expression profiles. While the single-cell techniques are extremely powerful and hold great promise, they are currently still labor intensive, have a high cost per cell, and, most importantly, do not provide information on spatial distribution of cell types in specific regions of the brain. We pr…
Brain-MGF: Multimodal Graph Fusion Network for EEG-fMRI Brain Connectivity Analysis Under Psilocybin
School of Information Technology, Monash University Malaysia（信息技术学院，墨尔本大学马来西亚分校）；Department of Data Science and AI, Faculty of Information Technology, Monash University（数据科学与人工智能系，信息科技学院，墨尔本大学）；Turner Institute for Brain and Mental Health, School of Psychological Sciences, Monash University（脑与心理健康研究所，心理学科学学院，墨尔本大学）
University of Catania（卡塔尼亚大学）；University of Central Florida（中佛罗里达大学）
This work presents a novel method of exploring human brain-visual representations, with a view towards replicating these processes in machines. The core idea is to learn plausible computational and biological representations by correlating human neural activity and natural images. Thus, we first propose a model, EEG-ChannelNet, to learn a brain manifold for EEG classification. After verifying that visual information can be extracted from EEG data, we introduce a multimodal approach that uses deep image and EEG encoders, trained in a siamese configuration, for learning a joint manifold that max…
Digital twin brain: a bridge between biological intelligence and artificial intelligence
Brainnetome Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所脑网络组研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Research Center for Augmented Intelligence, Zhejiang Lab（之江实验室增强智能研究中心）
