Recent studies have shown that multi-modeling methods can provide new insights into the analysis of brain components that are not possible when each modality is acquired separately. The joint representations of different modalities is a robust model to analyze simultaneously acquired electroencephalography and functional magnetic resonance imaging (EEG-fMRI). Advances in precision instruments have given us the ability to observe the spatiotemporal neural dynamics of the human brain through non-invasive neuroimaging techniques such as EEG & fMRI. Nonlinear fusion methods of streams can extract effective brain components in different dimensions of temporal and spatial. Graph-based analyzes, which have many similarities to brain structure, can overcome the complexities of brain mapping analysis. Throughout, we outline the correlations of several different media in time shifts from one source with graph-based and deep learning methods. Determining overlaps can provide a new perspective for diagnosing functional changes in neuroplasticity studies.
Higher School of Economics（高等经济学院）；Artificial Intelligence Research Institute, AIRI（人工智能研究所（AIRI））
The access to activity of subcortical structures offers unique opportunity for building intention dependent brain-computer interfaces, renders abundant options for exploring a broad range of cognitive phenomena in the realm of affective neuroscience including complex decision making processes and the eternal free-will dilemma and facilitates diagnostics of a range of neurological deceases. So far this was possible only using bulky, expensive and immobile fMRI equipment. Here we present an interpretable domain grounded solution to recover the activity of several subcortical regions from the mul…
Simultaneously recorded electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) can be used to non-invasively measure the spatiotemporal dynamics of the human brain. One challenge is dealing with the artifacts that each modality introduces into the other when the two are recorded concurrently, for example the ballistocardiogram (BCG). We conducted a preliminary comparison of three different MR compatible EEG recording systems and assessed their performance in terms of single-trial classification of the EEG when simultaneously collecting fMRI. We found tradeoffs across all…
This paper asks whether integrating multimodal EEG and fMRI data offers a better characterisation of functional brain architectures than either modality alone. This evaluation rests upon a dynamic causal model that generates both EEG and fMRI data from the same neuronal dynamics. We introduce the use of Bayesian fusion to provide informative (empirical) neuronal priors - derived from dynamic causal modelling (DCM) of EEG data - for subsequent DCM of fMRI data. To illustrate this procedure, we generated synthetic EEG and fMRI timeseries for a mismatch negativity (or auditory oddball) paradigm, …
Simultaneous EEG-fMRI is a multi-modal neuroimaging technique that provides complementary spatial and temporal resolution. Challenging has been developing principled and interpretable approaches for fusing the modalities, specifically approaches enabling inference of latent source spaces representative of neural activity. In this paper, we address this inference problem within the framework of transcoding -- mapping from a specific encoding (modality) to a decoding (the latent source space) and then encoding the latent source space to the other modality. Specifically, we develop a symmetric me…
Simultaneous real-time fMRI and EEG neurofeedback (rtfMRI-EEG-nf) is an emerging neuromodulation approach, that enables simultaneous volitional regulation of both hemodynamic (BOLD fMRI) and electrophysiological (EEG) brain activities. Here we report the first application of rtfMRI-EEG-nf for emotion self-regulation training in patients with major depressive disorder (MDD). In this proof-of-concept study, MDD patients in the experimental group (n=16) used rtfMRI-EEG-nf during a happy emotion induction task to simultaneously upregulate two fMRI and two EEG activity measures relevant to MDD. The…
Neuroimaging modalities such as functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provide information about neurological functions in complementary spatiotemporal resolutions; therefore, fusion of these modalities is expected to provide better understanding of brain activity. In this paper, we jointly analyze fMRI and multi-channel EEG signals collected during an auditory oddball task with the goal of capturing brain activity patterns that differ between patients with schizophrenia and healthy controls. Rather than selecting a single electrode or matricizing the thi…
Objective: The concurrent recording of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) is a technique that has received much attention due to its potential for combined high temporal and spatial resolution. However, the ballistocardiogram (BCG), a large-amplitude artifact caused by cardiac induced movement contaminates the EEG during EEG-fMRI recordings. Removal of BCG in software has generally made use of linear decompositions of the corrupted EEG. This is not ideal as the BCG signal is non-stationary and propagates in a manner which is non-linearly dependent on …
Recently, we reported an emotion self-regulation study (Zotev et al., 2020), in which patients with major depressive disorder (MDD) used simultaneous real-time fMRI and EEG neurofeedback (rtfMRI-EEG-nf) to upregulate two fMRI and two EEG activity measures, relevant to MDD. The target measures included fMRI activities of the left amygdala and left rostral anterior cingulate cortex, and frontal EEG asymmetries in the alpha band (FAA) and high-beta band (FBA). Here we apply the exact low resolution brain electromagnetic tomography (eLORETA) to investigate EEG source activities during the rtfMRI-E…
Human memory -- the learning of new information involves changes at the synaptic level between neurons dedicated for storage of in-formation. Generally, memory is classified as Long-Term Memory and Short-Term Memory. The various types of the memory and their disorder are widely studied using neuroimaging techniques like Electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI). Brain is effectively occupied with the capabilities of learning, retention and recall. The brain regions (pre-frontal cortex, associated hippocampus cortices and their interactions with other lobes) …
Simultaneous EEG-fMRI is a multi-modal neuroimaging technique that provides complementary spatial and temporal resolution for inferring a latent source space of neural activity. In this paper we address this inference problem within the framework of transcoding -- mapping from a specific encoding (modality) to a decoding (the latent source space) and then encoding the latent source space to the other modality. Specifically, we develop a symmetric method consisting of a cyclic convolutional transcoder that transcodes EEG to fMRI and vice versa. Without any prior knowledge of either the hemodyna…
EEG-to-fMRI synthesis of task-evoked and spontaneous brain activity: addressing issues of statistical significance and generalizability
Institute for Systems and Robotics – Lisboa and Department of Bioengineering, Instituto Superior Técnico, Universidade de Lisboa, Portugal（里斯本系统与机器人研究所及生物工程系，里斯本技术学院，里斯本大学，葡萄牙）；Electrical and Computer Engineering Department, Carnegie Mellon University（电气与计算机工程系，卡内基梅隆大学）
Neurofeedback is a promising approach for non-invasive modulation of human brain activity with applications for treatment of mental disorders and enhancement of brain performance. Neurofeedback techniques are commonly based on either electroencephalography (EEG) or real-time functional magnetic resonance imaging (rtfMRI). Advances in simultaneous EEG-fMRI have made it possible to combine the two approaches. Here we report the first implementation of simultaneous multimodal rtfMRI and EEG neurofeedback (rtfMRI-EEG-nf). It is based on a novel system for real-time integration of simultaneous rtfM…
KU Leuven（鲁汶大学）；Leuven Brain Institute（鲁汶脑科学研究所）；Maastricht UMC+（马斯特里赫特大学医学中心+）；University Hospitals Leuven（鲁汶大学医院）；Delft University of Technology（代尔夫特理工大学）
EEG-correlated fMRI analysis is widely used to detect regional blood oxygen level dependent fluctuations that are significantly synchronized to interictal epileptic discharges, which can provide evidence for localizing the ictal onset zone. However, such an asymmetrical, mass-univariate approach cannot capture the inherent, higher order structure in the EEG data, nor multivariate relations in the fMRI data, and it is nontrivial to accurately handle varying neurovascular coupling over patients and brain regions. We aim to overcome these drawbacks in a data-driven manner by means of a novel stru…
Univ Rennes（雷恩大学）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；University Hospital of Rennes（雷恩大学医院）
Simultaneous EEG/fMRI acquisition allows to measure brain activity at high spatial-temporal resolution. The localisation of EEG sources depends on several parameters including the position of the electrodes on the scalp. The position of the MR electrodes during its acquisitions is obtained with the use of the UTE sequence allowing their visualisation. The retrieval of the electrodes consists in obtaining the volume where the electrodes are located by applying a sphere detection algorithm. We detect around 90% of electrodes for each subject, and our UTE-based electrode detection showed an avera…
Real-time fMRI neurofeedback (rtfMRI-nf) is an emerging approach for studies and novel treatments of major depressive disorder (MDD). EEG performed simultaneously with an rtfMRI-nf procedure allows an independent evaluation of rtfMRI-nf brain modulation effects. Frontal EEG asymmetry in the alpha band is a widely used measure of emotion and motivation that shows profound changes in depression. However, it has never been directly related to simultaneously acquired fMRI data. We report the first study investigating electrophysiological correlates of the rtfMRI-nf procedure, by combining rtfMRI-n…
KU Leuven（鲁汶大学）；University of Piraeus（比雷埃夫斯大学）；KU Leuven Kulak（鲁汶大学科特赖克校区）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Data fusion refers to the joint analysis of multiple datasets which provide complementary views of the same task. In this preprint, the problem of jointly analyzing electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) data is considered. Jointly analyzing EEG and fMRI measurements is highly beneficial for studying brain function because these modalities have complementary spatiotemporal resolution: EEG offers good temporal resolution while fMRI is better in its spatial resolution. The fusion methods reported so far ignore the underlying multi-way nature of the data in …
INESC-ID（里斯本系统与计算机工程研究所）；Instituto Superior Técnico（高等技术学院）
Topographical structures represent connections between entities and provide a comprehensive design of complex systems. Currently these structures are used to discover correlates of neuronal and haemodynamical activity. In this work, we incorporate them with neural processing techniques to perform regression, using electrophysiological activity to retrieve haemodynamics. To this end, we use Fourier features, attention mechanisms, shared space between modalities and incorporation of style in the latent representation. By combining these techniques, we propose several models that significantly ou…
Brain-MGF: Multimodal Graph Fusion Network for EEG-fMRI Brain Connectivity Analysis Under Psilocybin
School of Information Technology, Monash University Malaysia（信息技术学院，墨尔本大学马来西亚分校）；Department of Data Science and AI, Faculty of Information Technology, Monash University（数据科学与人工智能系，信息科技学院，墨尔本大学）；Turner Institute for Brain and Mental Health, School of Psychological Sciences, Monash University（脑与心理健康研究所，心理学科学学院，墨尔本大学）
Neurofeedback is a non-invasive brain training with long-term medical and non-medical applications. Despite the existence of several emotion regulation studies using neurofeedback, further investigation is needed to understand interactions of the brain regions involved in the process. We implemented EEG neurofeedback with simultaneous fMRI using a modified happiness-inducing task through autobiographical memories to upregulate positive emotion. The results showed increased activity of prefrontal, occipital, parietal, and limbic regions and increased functional connectivity between prefrontal, …
Real-time fMRI neurofeedback (rtfMRI-nf) with simultaneous EEG allows volitional modulation of BOLD activity of target brain regions and investigation of related electrophysiological activity. We applied this approach to study correlations between thalamic BOLD activity and alpha EEG rhythm. Healthy volunteers in the experimental group (EG, n=15) learned to upregulate BOLD activity of the target region consisting of the mediodorsal (MD) and anterior (AN) thalamic nuclei using the rtfMRI-nf during retrieval of happy autobiographical memories. Healthy subjects in the control group (CG, n=14) wer…
Posttraumatic stress disorder (PTSD) is a chronic and disabling neuropsychiatric disorder characterized by insufficient top-down modulation of the amygdala activity by the prefrontal cortex. Real-time fMRI neurofeedback (rtfMRI-nf) is an emerging method with potential for modifying the amygdala-prefrontal interactions. We report the first controlled emotion self-regulation study in veterans with combat-related PTSD utilizing rtfMRI-nf of the amygdala activity. PTSD patients in the experimental group (EG, n=20) learned to upregulate BOLD activity of the left amygdala (LA) using rtfMRI-nf during…
The brain is intrinsically organized into large-scale networks that constantly re-organize on multiple timescales, even when the brain is at rest. The timing of these dynamics is crucial for sensation, perception, cognition and ultimately consciousness, but the underlying dynamics governing the constant reorganization and switching between networks are not yet well understood. Functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provide anatomical and temporal information about the resting-state networks (RSNs), respectively. EEG microstates are brief periods of stable…
Dartmouth College（达特茅斯学院）；University of Magdeburg（马格德堡大学）；Rutgers（罗格斯大学）；Carl von Ossietzky University Oldenburg（卡尔·冯·奥西茨基大学奥尔登堡分校）
Localizing neuronal activity in the brain, both in time and in space, is a central challenge to advance the understanding of brain function. Because of the inability of any single neuroimaging techniques to cover all aspects at once, there is a growing interest to combine signals from multiple modalities in order to benefit from the advantages of each acquisition method. Due to the complexity and unknown parameterization of any suggested complete model of BOLD response in functional magnetic resonance imaging (fMRI), the development of a reliable ultimate fusion approach remains difficult. But…
Estimated connectomes by the means of neuroimaging techniques have enriched our knowledge of the organizational properties of the brain leading to the development of network-based clinical diagnostics. Unfortunately, to date, many of those network-based clinical diagnostics tools, based on the mere description of isolated instances of observed connectomes are noisy estimates of the true connectivity network. Modeling brain connectivity networks is therefore important to better explain the functional organization of the brain and allow inference of specific brain properties. In this report, we …
Automated classification methods for disease diagnosis are currently in the limelight, especially for imaging data. Classification does not fully meet a clinician's needs, however: in order to combine the results of multiple tests and decide on a course of treatment, a clinician needs the likelihood of a given health condition rather than binary classification yielded by such methods. We illustrate how likelihoods can be derived step by step from first principles and approximations, and how they can be assessed and selected, illustrating our approach using fMRI data from a publicly available d…
CATD: Unified Representation Learning for EEG-to-fMRI Cross-Modal Generation
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Uppsala University（乌普萨拉大学）；King Fahd University of Petroleum and Minerals（法赫德国王石油与矿业大学）；Maebashi Institute of Technology（前桥工业大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）；Shenzhen University（深圳大学）
Burst suppression is an electroencephalography (EEG) pattern associated with profoundly inactivated brain states characterized by cerebral metabolic depression. Its distinctive feature is alternation between short temporal segments of near-isoelectric inactivity (suppressions) and relatively high-voltage activity (bursts). Prior modeling studies suggest that burst-suppression EEG is a manifestation of two alternating brain states associated with consumption (during a burst) and production (during a suppression) of adenosine triphosphate (ATP). This finding motivates us to infer latent states c…
Integration of Simultaneous Resting-State Electroencephalography, Functional Magnetic Resonance Imaging, and Eye-Tracker Methods to Determine and Verify Electroencephalography Vigilance Measure
Spontaneous fluctuations of the BOLD (Blood Oxygen Level-Dependent) signal, measured with fMRI (functional Magnetic Resonance Imaging), display a rich and neurobiologically relevant functional connectivity structure. This structure is usually revealed using time averaging methods, which prevent the detection of functional connectivity changes over time. In this work we studied the electrophysiological correlates of dynamical BOLD functional connectivity fluctuations, by means of long (approx. 50 min) joint electroencephalographic (EEG) and fMRI recordings. We identified widespread positive and…
High-resolution functional imaging is providing increasingly rich measurements of brain activity in animals and humans. A major challenge is to leverage such data to gain insight into the brain's computational mechanisms. The first step is to define candidate brain-computational models (BCMs) that can perform the behavioural task in question. We would then like to infer, which of the candidate BCMs best accounts for measured brain-activity data. Here we describe a method that complements each BCM by a measurement model (MM), which simulates the way the brain-activity measurements reflect neuro…
The University of Tokyo（东京大学）；Princeton University（普林斯顿大学）；Facebook Reality Labs（Facebook现实实验室）；Columbia University（哥伦比亚大学）；Intel Corporation（英特尔公司）
With the wide adoption of functional magnetic resonance imaging (fMRI) by cognitive neuroscience researchers, large volumes of brain imaging data have been accumulated in recent years. Aggregating these data to derive scientific insights often faces the challenge that fMRI data are high-dimensional, heterogeneous across people, and noisy. These challenges demand the development of computational tools that are tailored both for the neuroscience questions and for the properties of the data. We review a few recently developed algorithms in various domains of fMRI research: fMRI in naturalistic ta…
Dynamic functional connectivity (FC) has in recent years become a topic of interest in the neuroimaging community. Several models and methods exist for both functional magnetic resonance imaging (fMRI) and electroencephalography (EEG), and the results point towards the conclusion that FC exhibits dynamic changes. The existing approaches modeling dynamic connectivity have primarily been based on time-windowing the data and k-means clustering. We propose a non-parametric generative model for dynamic FC in fMRI that does not rely on specifying window lengths and number of dynamic states. Rooted i…
A large dataset of human EEG responses to short naturalistic videos for studying dynamic visual event processing
Freie Universität Berlin（柏林自由大学）；Charité – Universitätsmedizin Berlin（柏林夏里特医科大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）；University of Amsterdam（阿姆斯特丹大学）
Contemporary neuroscience has embraced network science to study the complex and self-organized structure of the human brain; one of the main outstanding issues is that of inferring from measure data, chiefly functional Magnetic Resonance Imaging (fMRI), the so-called effective connectivity in brain networks, that is the existing interactions among neuronal populations. This inverse problem is complicated by the fact that the BOLD (Blood Oxygenation Level Dependent) signal measured by fMRI represent a dynamic and nonlinear transformation (the hemodynamic response) of neuronal activity. In this …
Information mapping is a popular application of Multivoxel Pattern Analysis (MVPA) to fMRI. Information maps are constructed using the so called searchlight method, where the spherical multivoxel neighborhood of every voxel (i.e., a searchlight) in the brain is evaluated for the presence of task-relevant response patterns. Despite their widespread use, information maps present several challenges for interpretation. One such challenge has to do with inferring the size and shape of a multivoxel pattern from its signature on the information map. To address this issue, we formally examined the geo…
The anatomical structure of the brain can be observed via non-invasive techniques such as diffusion imaging. However, these are imperfect because they miss connections that are actually known to exist, especially long range inter-hemispheric ones. In this paper we formulate the inverse problem of inferring the structural connectivity of brain networks from experimentally observed functional connectivity via functional Magnetic Resonance Imaging (fMRI), by formulating it as a convex optimization problem. We show that structural connectivity can be modeled as an optimal sparse representation der…
Université de Montréal（蒙特利尔大学）；Mila – Quebec Artificial Intelligence Institute（米拉-魁北克人工智能研究所）；IMT Atlantique（IMT大西洋国立高等矿业电信学校）；IVADO, Université de Montréal（蒙特利尔大学IVADO研究所）；Hôpital du Sacré-Cœur de Montréal, Santé Québec Nord-de-l’Île-de-Montréal - Universitaire（蒙特利尔圣心医院，魁北克健康部蒙特利尔岛北-大学网络）；Faculty of Medicine, Université de Montréal（蒙特利尔大学医学院）
Universiti Teknologi Malaysia（马来西亚理工大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
We exploit altered patterns in brain functional connectivity as features for automatic discriminative analysis of neuropsychiatric patients. Deep learning methods have been introduced to functional network classification only very recently for fMRI, and the proposed architectures essentially focused on a single type of connectivity measure. We propose a deep convolutional neural network (CNN) framework for classification of electroencephalogram (EEG)-derived brain connectome in schizophrenia (SZ). To capture complementary aspects of disrupted connectivity in SZ, we explore combination of vario…
Applications of neuroimaging methods have substantially contributed to the scientific understanding of human factors during driving by providing a deeper insight into the neuro-cognitive aspects of driver brain. This has been achieved by conducting simulated (and occasionally, field) driving experiments while collecting driver brain signals of certain types. Here, this sector of studies is comprehensively reviewed at both macro and micro scales. Different themes of neuroimaging driving behaviour research are identified and the findings within each theme are synthesised. The surveyed literature…
Scaling temporal dynamics in functional MRI (fMRI) signals have been evidenced for a decade as intrinsic characteristics of ongoing brain activity (Zarahn et al., 1997). Recently, scaling properties were shown to fluctuate across brain networks and to be modulated between rest and task (He, 2011): notably, Hurst exponent, quantifying long memory, decreases under task in activating and deactivating brain regions. In most cases, such results were obtained: First, from univariate (voxelwise or regionwise) analysis, hence focusing on specific cognitive systems such as Resting-State Networks (RSNs)…
Inferring strength and direction of interactions from electroencephalographic (EEG) recordings is of crucial importance to improve our understanding of dynamical interdependencies underlying various physiologic and pathophysiologic conditions in the human epileptic brain. We here use approaches from symbolic analysis to investigate---in a time-resolved manner---weighted and directed, short- to long-ranged interactions between various brain regions constituting the epileptic network. Our observations point to complex spatial-temporal interdependencies underlying the epileptic process and their …
Objective Brain Age Index (BAI), calculated from sleep electroencephalography (EEG), has been proposed as a biomarker of brain health. This study quantifies night-to-night variability of BAI and establishes probability thresholds for inferring underlying brain pathology based on a patient's BAI. Methods 86 patients with multiple nights of consecutive EEG recordings were selected from Epilepsy Monitoring Unit patients whose EEGs reported as being within normal limits. BAI was calculated for each 12-hour segment of patient data using a previously described algorithm, and night-to-night variabili…
The brain computer interface (BCI) systems are utilized for transferring information among humans and computers by analyzing electroencephalogram (EEG) recordings.The process of mentally previewing a motor movement without generating the corporal output can be described as motor imagery (MI).In this emerging research field, the number of commands is also limited in relation to the number of MI tasks; in the current literature, mostly two or four commands (classes) are studied. As a solution to this problem, it is recommended to use mental tasks as well as MI tasks. Unfortunately, the use of th…
Conventional neuroimaging analyses have revealed the computational specificity of localized brain regions, exploiting the power of the subtraction technique in fMRI and event-related potential analyses in EEG. Moving beyond this convention, many researchers have begun exploring network-based neurodynamics and coordination between brain regions as a function of behavioral parameters or environmental statistics; however, most approaches average evoked activity across the experimental session to study task-dependent networks. Here, we examined on-going oscillatory activity and use a methodology t…
Multivariate pattern analysis (MVPA) or brain decoding methods have become standard practice in analysing fMRI data. Although decoding methods have been extensively applied in Brain Computing Interfaces (BCI), these methods have only recently been applied to time-series neuroimaging data such as MEG and EEG to address experimental questions in Cognitive Neuroscience. In a tutorial-style review, we describe a broad set of options to inform future time-series decoding studies from a Cognitive Neuroscience perspective. Using example MEG data, we illustrate the effects that different options in th…
