Simultaneously recorded electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) can be used to non-invasively measure the spatiotemporal dynamics of the human brain. One challenge is dealing with the artifacts that each modality introduces into the other when the two are recorded concurrently, for example the ballistocardiogram (BCG). We conducted a preliminary comparison of three different MR compatible EEG recording systems and assessed their performance in terms of single-trial classification of the EEG when simultaneously collecting fMRI. We found tradeoffs across all three systems, for example varied ease of setup and improved classification accuracy with reference electrodes (REF) but not for pulse artifact subtraction (PAS) or reference layer adaptive filtering (RLAF).
Neuroimaging modalities such as functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provide information about neurological functions in complementary spatiotemporal resolutions; therefore, fusion of these modalities is expected to provide better understanding of brain activity. In this paper, we jointly analyze fMRI and multi-channel EEG signals collected during an auditory oddball task with the goal of capturing brain activity patterns that differ between patients with schizophrenia and healthy controls. Rather than selecting a single electrode or matricizing the thi…
Advances on signal, image and video generation underly major breakthroughs on generative medical imaging tasks, including Brain Image Synthesis. Still, the extent to which functional Magnetic Ressonance Imaging (fMRI) can be mapped from the brain electrophysiology remains largely unexplored. This work provides the first comprehensive view on how to use state-of-the-art principles from Neural Processing to synthesize fMRI data from electroencephalographic (EEG) data. Given the distinct spatiotemporal nature of haemodynamic and electrophysiological signals, this problem is formulated as the task…
This paper asks whether integrating multimodal EEG and fMRI data offers a better characterisation of functional brain architectures than either modality alone. This evaluation rests upon a dynamic causal model that generates both EEG and fMRI data from the same neuronal dynamics. We introduce the use of Bayesian fusion to provide informative (empirical) neuronal priors - derived from dynamic causal modelling (DCM) of EEG data - for subsequent DCM of fMRI data. To illustrate this procedure, we generated synthetic EEG and fMRI timeseries for a mismatch negativity (or auditory oddball) paradigm, …
Simultaneous real-time fMRI and EEG neurofeedback (rtfMRI-EEG-nf) is an emerging neuromodulation approach, that enables simultaneous volitional regulation of both hemodynamic (BOLD fMRI) and electrophysiological (EEG) brain activities. Here we report the first application of rtfMRI-EEG-nf for emotion self-regulation training in patients with major depressive disorder (MDD). In this proof-of-concept study, MDD patients in the experimental group (n=16) used rtfMRI-EEG-nf during a happy emotion induction task to simultaneously upregulate two fMRI and two EEG activity measures relevant to MDD. The…
Higher School of Economics（高等经济学院）；Artificial Intelligence Research Institute, AIRI（人工智能研究所（AIRI））
The access to activity of subcortical structures offers unique opportunity for building intention dependent brain-computer interfaces, renders abundant options for exploring a broad range of cognitive phenomena in the realm of affective neuroscience including complex decision making processes and the eternal free-will dilemma and facilitates diagnostics of a range of neurological deceases. So far this was possible only using bulky, expensive and immobile fMRI equipment. Here we present an interpretable domain grounded solution to recover the activity of several subcortical regions from the mul…
Objective: The concurrent recording of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) is a technique that has received much attention due to its potential for combined high temporal and spatial resolution. However, the ballistocardiogram (BCG), a large-amplitude artifact caused by cardiac induced movement contaminates the EEG during EEG-fMRI recordings. Removal of BCG in software has generally made use of linear decompositions of the corrupted EEG. This is not ideal as the BCG signal is non-stationary and propagates in a manner which is non-linearly dependent on …
Recently, we reported an emotion self-regulation study (Zotev et al., 2020), in which patients with major depressive disorder (MDD) used simultaneous real-time fMRI and EEG neurofeedback (rtfMRI-EEG-nf) to upregulate two fMRI and two EEG activity measures, relevant to MDD. The target measures included fMRI activities of the left amygdala and left rostral anterior cingulate cortex, and frontal EEG asymmetries in the alpha band (FAA) and high-beta band (FBA). Here we apply the exact low resolution brain electromagnetic tomography (eLORETA) to investigate EEG source activities during the rtfMRI-E…
Human memory -- the learning of new information involves changes at the synaptic level between neurons dedicated for storage of in-formation. Generally, memory is classified as Long-Term Memory and Short-Term Memory. The various types of the memory and their disorder are widely studied using neuroimaging techniques like Electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI). Brain is effectively occupied with the capabilities of learning, retention and recall. The brain regions (pre-frontal cortex, associated hippocampus cortices and their interactions with other lobes) …
Simultaneous EEG-fMRI is a multi-modal neuroimaging technique that provides complementary spatial and temporal resolution for inferring a latent source space of neural activity. In this paper we address this inference problem within the framework of transcoding -- mapping from a specific encoding (modality) to a decoding (the latent source space) and then encoding the latent source space to the other modality. Specifically, we develop a symmetric method consisting of a cyclic convolutional transcoder that transcodes EEG to fMRI and vice versa. Without any prior knowledge of either the hemodyna…
KU Leuven（鲁汶大学）；University of Piraeus（比雷埃夫斯大学）；KU Leuven Kulak（鲁汶大学科特赖克校区）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Data fusion refers to the joint analysis of multiple datasets which provide complementary views of the same task. In this preprint, the problem of jointly analyzing electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) data is considered. Jointly analyzing EEG and fMRI measurements is highly beneficial for studying brain function because these modalities have complementary spatiotemporal resolution: EEG offers good temporal resolution while fMRI is better in its spatial resolution. The fusion methods reported so far ignore the underlying multi-way nature of the data in …
EEG-to-fMRI synthesis of task-evoked and spontaneous brain activity: addressing issues of statistical significance and generalizability
Institute for Systems and Robotics – Lisboa and Department of Bioengineering, Instituto Superior Técnico, Universidade de Lisboa, Portugal（里斯本系统与机器人研究所及生物工程系，里斯本技术学院，里斯本大学，葡萄牙）；Electrical and Computer Engineering Department, Carnegie Mellon University（电气与计算机工程系，卡内基梅隆大学）
Recent studies have shown that multi-modeling methods can provide new insights into the analysis of brain components that are not possible when each modality is acquired separately. The joint representations of different modalities is a robust model to analyze simultaneously acquired electroencephalography and functional magnetic resonance imaging (EEG-fMRI). Advances in precision instruments have given us the ability to observe the spatiotemporal neural dynamics of the human brain through non-invasive neuroimaging techniques such as EEG & fMRI. Nonlinear fusion methods of streams can extract …
Neurofeedback is a promising approach for non-invasive modulation of human brain activity with applications for treatment of mental disorders and enhancement of brain performance. Neurofeedback techniques are commonly based on either electroencephalography (EEG) or real-time functional magnetic resonance imaging (rtfMRI). Advances in simultaneous EEG-fMRI have made it possible to combine the two approaches. Here we report the first implementation of simultaneous multimodal rtfMRI and EEG neurofeedback (rtfMRI-EEG-nf). It is based on a novel system for real-time integration of simultaneous rtfM…
KU Leuven（鲁汶大学）；Leuven Brain Institute（鲁汶脑科学研究所）；Maastricht UMC+（马斯特里赫特大学医学中心+）；University Hospitals Leuven（鲁汶大学医院）；Delft University of Technology（代尔夫特理工大学）
EEG-correlated fMRI analysis is widely used to detect regional blood oxygen level dependent fluctuations that are significantly synchronized to interictal epileptic discharges, which can provide evidence for localizing the ictal onset zone. However, such an asymmetrical, mass-univariate approach cannot capture the inherent, higher order structure in the EEG data, nor multivariate relations in the fMRI data, and it is nontrivial to accurately handle varying neurovascular coupling over patients and brain regions. We aim to overcome these drawbacks in a data-driven manner by means of a novel stru…
Univ Rennes（雷恩大学）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；University Hospital of Rennes（雷恩大学医院）
Simultaneous EEG/fMRI acquisition allows to measure brain activity at high spatial-temporal resolution. The localisation of EEG sources depends on several parameters including the position of the electrodes on the scalp. The position of the MR electrodes during its acquisitions is obtained with the use of the UTE sequence allowing their visualisation. The retrieval of the electrodes consists in obtaining the volume where the electrodes are located by applying a sphere detection algorithm. We detect around 90% of electrodes for each subject, and our UTE-based electrode detection showed an avera…
Simultaneous EEG-fMRI is a multi-modal neuroimaging technique that provides complementary spatial and temporal resolution. Challenging has been developing principled and interpretable approaches for fusing the modalities, specifically approaches enabling inference of latent source spaces representative of neural activity. In this paper, we address this inference problem within the framework of transcoding -- mapping from a specific encoding (modality) to a decoding (the latent source space) and then encoding the latent source space to the other modality. Specifically, we develop a symmetric me…
Real-time fMRI neurofeedback (rtfMRI-nf) is an emerging approach for studies and novel treatments of major depressive disorder (MDD). EEG performed simultaneously with an rtfMRI-nf procedure allows an independent evaluation of rtfMRI-nf brain modulation effects. Frontal EEG asymmetry in the alpha band is a widely used measure of emotion and motivation that shows profound changes in depression. However, it has never been directly related to simultaneously acquired fMRI data. We report the first study investigating electrophysiological correlates of the rtfMRI-nf procedure, by combining rtfMRI-n…
INESC-ID（里斯本系统与计算机工程研究所）；Instituto Superior Técnico（高等技术学院）
Topographical structures represent connections between entities and provide a comprehensive design of complex systems. Currently these structures are used to discover correlates of neuronal and haemodynamical activity. In this work, we incorporate them with neural processing techniques to perform regression, using electrophysiological activity to retrieve haemodynamics. To this end, we use Fourier features, attention mechanisms, shared space between modalities and incorporation of style in the latent representation. By combining these techniques, we propose several models that significantly ou…
Brain-MGF: Multimodal Graph Fusion Network for EEG-fMRI Brain Connectivity Analysis Under Psilocybin
School of Information Technology, Monash University Malaysia（信息技术学院，墨尔本大学马来西亚分校）；Department of Data Science and AI, Faculty of Information Technology, Monash University（数据科学与人工智能系，信息科技学院，墨尔本大学）；Turner Institute for Brain and Mental Health, School of Psychological Sciences, Monash University（脑与心理健康研究所，心理学科学学院，墨尔本大学）
Neurofeedback is a non-invasive brain training with long-term medical and non-medical applications. Despite the existence of several emotion regulation studies using neurofeedback, further investigation is needed to understand interactions of the brain regions involved in the process. We implemented EEG neurofeedback with simultaneous fMRI using a modified happiness-inducing task through autobiographical memories to upregulate positive emotion. The results showed increased activity of prefrontal, occipital, parietal, and limbic regions and increased functional connectivity between prefrontal, …
Real-time fMRI neurofeedback (rtfMRI-nf) with simultaneous EEG allows volitional modulation of BOLD activity of target brain regions and investigation of related electrophysiological activity. We applied this approach to study correlations between thalamic BOLD activity and alpha EEG rhythm. Healthy volunteers in the experimental group (EG, n=15) learned to upregulate BOLD activity of the target region consisting of the mediodorsal (MD) and anterior (AN) thalamic nuclei using the rtfMRI-nf during retrieval of happy autobiographical memories. Healthy subjects in the control group (CG, n=14) wer…
The Weizmann Institute of Science（魏茨曼科学研究所）；Zuckerman Institute（朱克曼研究所）；Columbia University（哥伦比亚大学）
Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder n…
By computerized analysis of cortical activity recorded via fMRI for pediatric epilepsy patients, we implement algorithmic localization of epileptic seizure focus within one of eight cortical lobes. Our innovative machine learning techniques involve intensive analysis of large matrices of mutual information coefficients between pairs of anatomically identified cortical regions. Drastic selection of pairs of regions with significant inter-connectivity provide efficient inputs for our Multi-Layer Perceptron (MLP) classifier. By imposing rigorous parameter parsimony to avoid over fitting we constr…
Posttraumatic stress disorder (PTSD) is a chronic and disabling neuropsychiatric disorder characterized by insufficient top-down modulation of the amygdala activity by the prefrontal cortex. Real-time fMRI neurofeedback (rtfMRI-nf) is an emerging method with potential for modifying the amygdala-prefrontal interactions. We report the first controlled emotion self-regulation study in veterans with combat-related PTSD utilizing rtfMRI-nf of the amygdala activity. PTSD patients in the experimental group (EG, n=20) learned to upregulate BOLD activity of the left amygdala (LA) using rtfMRI-nf during…
Objective: To quantify the effect of inhaled 5% carbon-dioxide/95% oxygen on EEG recordings from patients in non-convulsive status epilepticus (NCSE). Methods: Five children of mixed aetiology in NCSE were given high flow of inhaled carbogen (5% carbon dioxide/95% oxygen) using a face mask for maximum 120s. EEG was recorded concurrently in all patients. The effects of inhaled carbogen on patient EEG recordings were investigated using band-power, functional connectivity and graph theory measures. Carbogen effect was quantified by measuring effect size (Cohen's d) between "before", "during" and …
The brain is intrinsically organized into large-scale networks that constantly re-organize on multiple timescales, even when the brain is at rest. The timing of these dynamics is crucial for sensation, perception, cognition and ultimately consciousness, but the underlying dynamics governing the constant reorganization and switching between networks are not yet well understood. Functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provide anatomical and temporal information about the resting-state networks (RSNs), respectively. EEG microstates are brief periods of stable…
This study demonstrates the capability of external signal recording into memory and the reproduction of memory trace of this pattern in EEG by direct AC electrical stimulation of rat cerebral cortex. Additionally, we examine shifts of the DC potential level related to these phenomena. We show that in the course of memory trace reproduction, consecutive phases of engram activation and relaxation are registered and accompanied by corresponding negative and positive DC shifts. The observed electrophysiological changes may reflect consecutive activation and inhibition phases of neural ensembles pa…
We want to analyse EEG recordings in order to investigate the phonemic categorization at a very early stage of auditory processing. This problem can be modelled by a supervised classification of functional data. Discrimination is explored via a logistic functional linear model, using a wavelet representation of the data. Different procedures are investigated, based on penalized likelihood and principal component reduction or partial least squares reduction.
Dartmouth College（达特茅斯学院）；University of Magdeburg（马格德堡大学）；Rutgers（罗格斯大学）；Carl von Ossietzky University Oldenburg（卡尔·冯·奥西茨基大学奥尔登堡分校）
Localizing neuronal activity in the brain, both in time and in space, is a central challenge to advance the understanding of brain function. Because of the inability of any single neuroimaging techniques to cover all aspects at once, there is a growing interest to combine signals from multiple modalities in order to benefit from the advantages of each acquisition method. Due to the complexity and unknown parameterization of any suggested complete model of BOLD response in functional magnetic resonance imaging (fMRI), the development of a reliable ultimate fusion approach remains difficult. But…
In the past few years, significant advancements were made in reconstruction of observed natural images from fMRI brain recordings using deep-learning tools. Here, for the first time, we show that dense 3D depth maps of observed 2D natural images can also be recovered directly from fMRI brain recordings. We use an off-the-shelf method to estimate the unknown depth maps of natural images. This is applied to both: (i) the small number of images presented to subjects in an fMRI scanner (images for which we have fMRI recordings - referred to as "paired" data), and (ii) a very large number of natura…
We performed simultaneous recordings of electroencephalography (EEG) from multiple students in a classroom, and measured the inter-subject correlation (ISC) of activity evoked by a common video stimulus. The neural reliability, as quantified by ISC, has been linked to engagement and attentional modulation in earlier studies that used high-grade equipment in laboratory settings. Here we reproduce many of the results from these studies using portable low-cost equipment, focusing on the robustness of using ISC for subjects experiencing naturalistic stimuli. The present data shows that stimulus-ev…
CATD: Unified Representation Learning for EEG-to-fMRI Cross-Modal Generation
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Uppsala University（乌普萨拉大学）；King Fahd University of Petroleum and Minerals（法赫德国王石油与矿业大学）；Maebashi Institute of Technology（前桥工业大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）；Shenzhen University（深圳大学）
The amount of power in different frequency bands of the electroencephalogram (EEG) carries information about the behavioral state of a subject. Hence, neurologists treating epileptic patients monitor the temporal evolution of the different bands. We propose a covariance-based method to detect and characterize epileptic seizures operating on the band-filtered EEG signal. The algorithm is unsupervised, and performs a principal component analysis of intra-cranial EEG recordings, detecting transient fluctuations of the power in each frequency band. Its simplicity makes it suitable for online imple…
Cardiff University（卡迪夫大学）；University of the West of England（西英格兰大学）
The classification of electroencephalography (EEG) signals is useful in a wide range of applications such as seizure detection/prediction, motor imagery classification, emotion classification and drug effects diagnosis, amongst others. With the large number of EEG channels acquired, it has become vital that efficient data-reduction methods are developed, with varying importance from one application to another. It is also important that online classification is achieved during EEG recording for many applications, to monitor changes as they happen. In this paper we introduce a method based on Mu…
The resistive or non-resistive nature of the extracellular space in the brain is still debated, and is an important issue for correctly modeling extracellular potentials. Here, we first show theoretically that if the medium is resistive, the frequency scaling should be the same for electroencephalogram (EEG) and magnetoencephalogram (MEG) signals at low frequencies (<10 Hz). To test this prediction, we analyzed the spectrum of simultaneous EEG and MEG measurements in four human subjects. The frequency scaling of EEG displays coherent variations across the brain, in general between 1/f and 1/f^…
In this viewpoint article, we discuss the electric properties of the medium around neurons, which are important to correctly interpret extracellular potentials or electric field effects in neural tissue. We focus on how these electric properties shape the frequency scaling of brain signals at different scales, such as intracellular recordings, the local field potential (LFP), the electroencephalogram (EEG) or the magnetoencephalogram (MEG). These signals display frequency-scaling properties which are not consistent with resistive media. The medium appears to exert a frequency filtering scaling…
The extent of intra-individual and inter-individual variability is an important factor in determining the statistical, and hence possibly clinical, significance of observed differences in the EEG. This study investigates the changes in classical quantitative EEG (qEEG) measures, as well as of parameters obtained by fitting frequency spectra to a continuum model of brain electrical activity. These parameters may have extra variability due to model selection and fitting. Besides estimating levels of intra-individual and inter-individual variability, we determine approximate time scales for chang…
We described driver behaviour and brain dynamics acquired from a 90-minute sustained-attention task in an immersive driving simulator. The data include 62 copies of 32 channel electroencephalography (EEG) data for 27 subjects that drove on a four lane highway and were asked to keep the car cruising in the centre of the lane. Lane departure events were randomly induced to make the car drift from the original cruising lane towards the left or right lane. A complete trial includes events with deviation onset, response onset, and response offset. The next trial, in which the subject has to drive b…
We introduce a mean-field model for analysing the dynamics of human consciousness. In particular, inspired by the Giulio Tononi's Integrated Information Theory and by the Max Tegmark's representation of consciousness, we study order-disorder phase transitions on Curie-Weiss models generated by processing EEG signals. The latter have been recorded on healthy individuals undergoing deep sedation. Then, we implement a machine learning tool for classifying mental states using, as input, the critical temperatures computed in the Curie-Weiss models. Results show that, by the proposed method, it is p…
Goal: Limited-channel EEG research in neonates is hindered by lack of open, accessible analytic tools. To overcome this limitation, we have created the Washington University- Neonatal EEG Analysis Toolbox (WU-NEAT), containing two of the most commonly used tools, provided in an open-source, clinically-validated package running within MATLAB. Methods: The first algorithm is the amplitude-integrated EEG (aEEG), which is generated by filtering, rectifying and time-compressing the original EEG recording, with subsequent semi-logarithmic display. The second algorithm is the spectral edge frequency …
$\textbf{Objective}$: To assess the validity of an automatic EEG arousal detection algorithm using large patient samples and different heterogeneous databases $\textbf{Methods}$: Automatic scorings were confronted with results from human expert scorers on a total of 2768 full-night PSG recordings obtained from two different databases. Of them, 472 recordings were obtained during clinical routine at our sleep center and were subdivided into two subgroups of 220 (HMC-S) and 252 (HMC-M) recordings each, attending to the procedure followed by the clinical expert during the visual review (semi-auto…
University of Basel（巴塞尔大学）；Hospital of the University of Basel（巴塞尔大学医院）
Recordings of electrical brain activity carry information about a person's cognitive health. For recording EEG signals, a very common setting is for a subject to be at rest with its eyes closed. Analysis of these recordings often involve a dimensionality reduction step in which electrodes are grouped into 10 or more regions (depending on the number of electrodes available). Then an average over each group is taken which serves as a feature in subsequent evaluation. Currently, the most prominent features used in clinical practice are based on spectral power densities. In our work we consider a …
Working Memory (WM) load is an important cognitive feature that is highly correlated with mental effort. Several neurological biomarkers such as theta power and mid-frontal activity show increased activity with increasing WM load. Such correlations often break down in cognitively impaired individuals, making WM load biomarkers a valuable tool for the detection of cognitive impairment. However, most studies have used a multi-channel EEG or an fMRI, which are not massively accessible. In the present study, we evaluate the ability of novel features extracted from a single-channel EEG located on t…
There are several protocols in the Electroencephalography (EEG) recording scenarios which produce various types of event-related potentials (ERP). P300 pattern is a well-known ERP which produced by auditory and visual oddball paradigm and BCI speller system. In this study, P300 and non-P300 separability are investigated in two scenarios including image recognition paradigm and BCI speller. Image recognition scenario is an experiment that examines the participants, knowledge about an image that shown to them before by analyzing the EEG signal recorded during the observing of that image as visua…
Université Paris-Saclay（巴黎萨克雷大学）；Inria（法国国家信息与自动化研究所）；CEA（法国原子能和替代能源委员会）；InteraXon Inc.（InteraXon公司）；Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）
Building machine learning models using EEG recorded outside of the laboratory setting requires methods robust to noisy data and randomly missing channels. This need is particularly great when working with sparse EEG montages (1-6 channels), often encountered in consumer-grade or mobile EEG devices. Neither classical machine learning models nor deep neural networks trained end-to-end on EEG are typically designed or tested for robustness to corruption, and especially to randomly missing channels. While some studies have proposed strategies for using data with missing channels, these approaches …
