Human memory -- the learning of new information involves changes at the synaptic level between neurons dedicated for storage of in-formation. Generally, memory is classified as Long-Term Memory and Short-Term Memory. The various types of the memory and their disorder are widely studied using neuroimaging techniques like Electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI). Brain is effectively occupied with the capabilities of learning, retention and recall. The brain regions (pre-frontal cortex, associated hippocampus cortices and their interactions with other lobes) involved in memory recall tasks focuses on understanding the memory retention and recall processes. However, due to highly complicated and dynamic mechanisms of brain, the specific regions where information may reside are not completely explored. In this research paper, recent memory literature using EEG and fMRI studies is reviewed to understand the memory retention and recall processes as well as the various brain regions associated with these processes. A number of stimuli which are reported in previous studies are evaluated and discussed. Furthermore, the challenges which are being faced by researchers in EEG and fMRI methodologies are also presented. Recommendations for the future research related to memory retention and recall are also discussed at the end.
Simultaneously recorded electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) can be used to non-invasively measure the spatiotemporal dynamics of the human brain. One challenge is dealing with the artifacts that each modality introduces into the other when the two are recorded concurrently, for example the ballistocardiogram (BCG). We conducted a preliminary comparison of three different MR compatible EEG recording systems and assessed their performance in terms of single-trial classification of the EEG when simultaneously collecting fMRI. We found tradeoffs across all…
This paper asks whether integrating multimodal EEG and fMRI data offers a better characterisation of functional brain architectures than either modality alone. This evaluation rests upon a dynamic causal model that generates both EEG and fMRI data from the same neuronal dynamics. We introduce the use of Bayesian fusion to provide informative (empirical) neuronal priors - derived from dynamic causal modelling (DCM) of EEG data - for subsequent DCM of fMRI data. To illustrate this procedure, we generated synthetic EEG and fMRI timeseries for a mismatch negativity (or auditory oddball) paradigm, …
Simultaneous real-time fMRI and EEG neurofeedback (rtfMRI-EEG-nf) is an emerging neuromodulation approach, that enables simultaneous volitional regulation of both hemodynamic (BOLD fMRI) and electrophysiological (EEG) brain activities. Here we report the first application of rtfMRI-EEG-nf for emotion self-regulation training in patients with major depressive disorder (MDD). In this proof-of-concept study, MDD patients in the experimental group (n=16) used rtfMRI-EEG-nf during a happy emotion induction task to simultaneously upregulate two fMRI and two EEG activity measures relevant to MDD. The…
Objective: The concurrent recording of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) is a technique that has received much attention due to its potential for combined high temporal and spatial resolution. However, the ballistocardiogram (BCG), a large-amplitude artifact caused by cardiac induced movement contaminates the EEG during EEG-fMRI recordings. Removal of BCG in software has generally made use of linear decompositions of the corrupted EEG. This is not ideal as the BCG signal is non-stationary and propagates in a manner which is non-linearly dependent on …
Recently, we reported an emotion self-regulation study (Zotev et al., 2020), in which patients with major depressive disorder (MDD) used simultaneous real-time fMRI and EEG neurofeedback (rtfMRI-EEG-nf) to upregulate two fMRI and two EEG activity measures, relevant to MDD. The target measures included fMRI activities of the left amygdala and left rostral anterior cingulate cortex, and frontal EEG asymmetries in the alpha band (FAA) and high-beta band (FBA). Here we apply the exact low resolution brain electromagnetic tomography (eLORETA) to investigate EEG source activities during the rtfMRI-E…
Neuroimaging modalities such as functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provide information about neurological functions in complementary spatiotemporal resolutions; therefore, fusion of these modalities is expected to provide better understanding of brain activity. In this paper, we jointly analyze fMRI and multi-channel EEG signals collected during an auditory oddball task with the goal of capturing brain activity patterns that differ between patients with schizophrenia and healthy controls. Rather than selecting a single electrode or matricizing the thi…
EEG-to-fMRI synthesis of task-evoked and spontaneous brain activity: addressing issues of statistical significance and generalizability
Institute for Systems and Robotics – Lisboa and Department of Bioengineering, Instituto Superior Técnico, Universidade de Lisboa, Portugal（里斯本系统与机器人研究所及生物工程系，里斯本技术学院，里斯本大学，葡萄牙）；Electrical and Computer Engineering Department, Carnegie Mellon University（电气与计算机工程系，卡内基梅隆大学）
Higher School of Economics（高等经济学院）；Artificial Intelligence Research Institute, AIRI（人工智能研究所（AIRI））
The access to activity of subcortical structures offers unique opportunity for building intention dependent brain-computer interfaces, renders abundant options for exploring a broad range of cognitive phenomena in the realm of affective neuroscience including complex decision making processes and the eternal free-will dilemma and facilitates diagnostics of a range of neurological deceases. So far this was possible only using bulky, expensive and immobile fMRI equipment. Here we present an interpretable domain grounded solution to recover the activity of several subcortical regions from the mul…
Neurofeedback is a non-invasive brain training with long-term medical and non-medical applications. Despite the existence of several emotion regulation studies using neurofeedback, further investigation is needed to understand interactions of the brain regions involved in the process. We implemented EEG neurofeedback with simultaneous fMRI using a modified happiness-inducing task through autobiographical memories to upregulate positive emotion. The results showed increased activity of prefrontal, occipital, parietal, and limbic regions and increased functional connectivity between prefrontal, …
Recent studies have shown that multi-modeling methods can provide new insights into the analysis of brain components that are not possible when each modality is acquired separately. The joint representations of different modalities is a robust model to analyze simultaneously acquired electroencephalography and functional magnetic resonance imaging (EEG-fMRI). Advances in precision instruments have given us the ability to observe the spatiotemporal neural dynamics of the human brain through non-invasive neuroimaging techniques such as EEG & fMRI. Nonlinear fusion methods of streams can extract …
Simultaneous EEG-fMRI is a multi-modal neuroimaging technique that provides complementary spatial and temporal resolution for inferring a latent source space of neural activity. In this paper we address this inference problem within the framework of transcoding -- mapping from a specific encoding (modality) to a decoding (the latent source space) and then encoding the latent source space to the other modality. Specifically, we develop a symmetric method consisting of a cyclic convolutional transcoder that transcodes EEG to fMRI and vice versa. Without any prior knowledge of either the hemodyna…
Neurofeedback is a promising approach for non-invasive modulation of human brain activity with applications for treatment of mental disorders and enhancement of brain performance. Neurofeedback techniques are commonly based on either electroencephalography (EEG) or real-time functional magnetic resonance imaging (rtfMRI). Advances in simultaneous EEG-fMRI have made it possible to combine the two approaches. Here we report the first implementation of simultaneous multimodal rtfMRI and EEG neurofeedback (rtfMRI-EEG-nf). It is based on a novel system for real-time integration of simultaneous rtfM…
KU Leuven（鲁汶大学）；Leuven Brain Institute（鲁汶脑科学研究所）；Maastricht UMC+（马斯特里赫特大学医学中心+）；University Hospitals Leuven（鲁汶大学医院）；Delft University of Technology（代尔夫特理工大学）
EEG-correlated fMRI analysis is widely used to detect regional blood oxygen level dependent fluctuations that are significantly synchronized to interictal epileptic discharges, which can provide evidence for localizing the ictal onset zone. However, such an asymmetrical, mass-univariate approach cannot capture the inherent, higher order structure in the EEG data, nor multivariate relations in the fMRI data, and it is nontrivial to accurately handle varying neurovascular coupling over patients and brain regions. We aim to overcome these drawbacks in a data-driven manner by means of a novel stru…
Univ Rennes（雷恩大学）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；University Hospital of Rennes（雷恩大学医院）
Simultaneous EEG/fMRI acquisition allows to measure brain activity at high spatial-temporal resolution. The localisation of EEG sources depends on several parameters including the position of the electrodes on the scalp. The position of the MR electrodes during its acquisitions is obtained with the use of the UTE sequence allowing their visualisation. The retrieval of the electrodes consists in obtaining the volume where the electrodes are located by applying a sphere detection algorithm. We detect around 90% of electrodes for each subject, and our UTE-based electrode detection showed an avera…
Simultaneous EEG-fMRI is a multi-modal neuroimaging technique that provides complementary spatial and temporal resolution. Challenging has been developing principled and interpretable approaches for fusing the modalities, specifically approaches enabling inference of latent source spaces representative of neural activity. In this paper, we address this inference problem within the framework of transcoding -- mapping from a specific encoding (modality) to a decoding (the latent source space) and then encoding the latent source space to the other modality. Specifically, we develop a symmetric me…
Real-time fMRI neurofeedback (rtfMRI-nf) is an emerging approach for studies and novel treatments of major depressive disorder (MDD). EEG performed simultaneously with an rtfMRI-nf procedure allows an independent evaluation of rtfMRI-nf brain modulation effects. Frontal EEG asymmetry in the alpha band is a widely used measure of emotion and motivation that shows profound changes in depression. However, it has never been directly related to simultaneously acquired fMRI data. We report the first study investigating electrophysiological correlates of the rtfMRI-nf procedure, by combining rtfMRI-n…
The brain is intrinsically organized into large-scale networks that constantly re-organize on multiple timescales, even when the brain is at rest. The timing of these dynamics is crucial for sensation, perception, cognition and ultimately consciousness, but the underlying dynamics governing the constant reorganization and switching between networks are not yet well understood. Functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provide anatomical and temporal information about the resting-state networks (RSNs), respectively. EEG microstates are brief periods of stable…
INESC-ID（里斯本系统与计算机工程研究所）；Instituto Superior Técnico（高等技术学院）
Topographical structures represent connections between entities and provide a comprehensive design of complex systems. Currently these structures are used to discover correlates of neuronal and haemodynamical activity. In this work, we incorporate them with neural processing techniques to perform regression, using electrophysiological activity to retrieve haemodynamics. To this end, we use Fourier features, attention mechanisms, shared space between modalities and incorporation of style in the latent representation. By combining these techniques, we propose several models that significantly ou…
Brain-MGF: Multimodal Graph Fusion Network for EEG-fMRI Brain Connectivity Analysis Under Psilocybin
School of Information Technology, Monash University Malaysia（信息技术学院，墨尔本大学马来西亚分校）；Department of Data Science and AI, Faculty of Information Technology, Monash University（数据科学与人工智能系，信息科技学院，墨尔本大学）；Turner Institute for Brain and Mental Health, School of Psychological Sciences, Monash University（脑与心理健康研究所，心理学科学学院，墨尔本大学）
KU Leuven（鲁汶大学）；University of Piraeus（比雷埃夫斯大学）；KU Leuven Kulak（鲁汶大学科特赖克校区）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Data fusion refers to the joint analysis of multiple datasets which provide complementary views of the same task. In this preprint, the problem of jointly analyzing electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) data is considered. Jointly analyzing EEG and fMRI measurements is highly beneficial for studying brain function because these modalities have complementary spatiotemporal resolution: EEG offers good temporal resolution while fMRI is better in its spatial resolution. The fusion methods reported so far ignore the underlying multi-way nature of the data in …
Real-time fMRI neurofeedback (rtfMRI-nf) with simultaneous EEG allows volitional modulation of BOLD activity of target brain regions and investigation of related electrophysiological activity. We applied this approach to study correlations between thalamic BOLD activity and alpha EEG rhythm. Healthy volunteers in the experimental group (EG, n=15) learned to upregulate BOLD activity of the target region consisting of the mediodorsal (MD) and anterior (AN) thalamic nuclei using the rtfMRI-nf during retrieval of happy autobiographical memories. Healthy subjects in the control group (CG, n=14) wer…
Posttraumatic stress disorder (PTSD) is a chronic and disabling neuropsychiatric disorder characterized by insufficient top-down modulation of the amygdala activity by the prefrontal cortex. Real-time fMRI neurofeedback (rtfMRI-nf) is an emerging method with potential for modifying the amygdala-prefrontal interactions. We report the first controlled emotion self-regulation study in veterans with combat-related PTSD utilizing rtfMRI-nf of the amygdala activity. PTSD patients in the experimental group (EG, n=20) learned to upregulate BOLD activity of the left amygdala (LA) using rtfMRI-nf during…
Dartmouth College（达特茅斯学院）；University of Magdeburg（马格德堡大学）；Rutgers（罗格斯大学）；Carl von Ossietzky University Oldenburg（卡尔·冯·奥西茨基大学奥尔登堡分校）
Localizing neuronal activity in the brain, both in time and in space, is a central challenge to advance the understanding of brain function. Because of the inability of any single neuroimaging techniques to cover all aspects at once, there is a growing interest to combine signals from multiple modalities in order to benefit from the advantages of each acquisition method. Due to the complexity and unknown parameterization of any suggested complete model of BOLD response in functional magnetic resonance imaging (fMRI), the development of a reliable ultimate fusion approach remains difficult. But…
CATD: Unified Representation Learning for EEG-to-fMRI Cross-Modal Generation
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Uppsala University（乌普萨拉大学）；King Fahd University of Petroleum and Minerals（法赫德国王石油与矿业大学）；Maebashi Institute of Technology（前桥工业大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）；Shenzhen University（深圳大学）
Working Memory (WM) load is an important cognitive feature that is highly correlated with mental effort. Several neurological biomarkers such as theta power and mid-frontal activity show increased activity with increasing WM load. Such correlations often break down in cognitively impaired individuals, making WM load biomarkers a valuable tool for the detection of cognitive impairment. However, most studies have used a multi-channel EEG or an fMRI, which are not massively accessible. In the present study, we evaluate the ability of novel features extracted from a single-channel EEG located on t…
A large dataset of human EEG responses to short naturalistic videos for studying dynamic visual event processing
Freie Universität Berlin（柏林自由大学）；Charité – Universitätsmedizin Berlin（柏林夏里特医科大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）；University of Amsterdam（阿姆斯特丹大学）
Working memory (WM) is a mechanism that temporarily stores and manipulates information in service of behavioral goals and is a highly dynamic process. Previous studies have considered decoding WM load using EEG but have not investigated the contribution of sequential information contained in the temporal patterns of the EEG data that can differentiate different WM loads. In our study, we develop a novel method of investigating the role of sequential information in the manipulation and storage of verbal information at various time scales and localize topographically the sources of the sequentia…
University of Belgrade（贝尔格莱德大学）；Amsterdam Health and Technology Institute（阿姆斯特丹健康与技术研究所）
In this paper, we aimed at reviewing several different approaches present today in the search for more accurate diagnostic and treatment management in mental healthcare. Our focus is on mood disorders, and in particular on the major depressive disorder (MDD). We are reviewing and discussing findings based on neuroimaging studies (MRI and fMRI) first to get the impression of the body of knowledge about the anatomical and functional differences in depression. Then, we are focusing on less expensive data-driven approach, applicable for everyday clinical practice, in particular, those based on ele…
Brain activity following stimulus presentation and during resting state are often the result of highly coordinated responses of large numbers of neurons both locally and globally. Coordinated activity of neurons can give rise to oscillations which are captured by electroencephalograms (EEG). In this paper, we examine EEGs as this is the primary data being used by our collaborators who are studying coordination of neuronal response during the execution of tasks such as learning, and memory formation, retention and retrieval. In this paper, we develop the spectral merger clustering (SMC) method …
Universiti Teknologi Malaysia（马来西亚理工大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
We exploit altered patterns in brain functional connectivity as features for automatic discriminative analysis of neuropsychiatric patients. Deep learning methods have been introduced to functional network classification only very recently for fMRI, and the proposed architectures essentially focused on a single type of connectivity measure. We propose a deep convolutional neural network (CNN) framework for classification of electroencephalogram (EEG)-derived brain connectome in schizophrenia (SZ). To capture complementary aspects of disrupted connectivity in SZ, we explore combination of vario…
Beyond Performance Scores: Directed Functional Connectivity as a Brain-Based Biomarker for Motor Skill Learning and Retention
Rensselaer Polytechnic Institute（伦斯勒理工学院）；University at Buffalo（布法罗大学）；University at Buffalo Jacobs School of Medicine and Biomedical Sciences（布法罗大学雅各布斯医学与生物医学科学学院）；Uniformed Services University of the Health Sciences（uniformed services health sciences university）；U.S. Army Futures Command, Combat Capabilities Development Command Soldier Center STTC（美国陆军未来司令部作战能力发展司令部士兵中心STTC）
The brain computer interface (BCI) systems are utilized for transferring information among humans and computers by analyzing electroencephalogram (EEG) recordings.The process of mentally previewing a motor movement without generating the corporal output can be described as motor imagery (MI).In this emerging research field, the number of commands is also limited in relation to the number of MI tasks; in the current literature, mostly two or four commands (classes) are studied. As a solution to this problem, it is recommended to use mental tasks as well as MI tasks. Unfortunately, the use of th…
Conventional neuroimaging analyses have revealed the computational specificity of localized brain regions, exploiting the power of the subtraction technique in fMRI and event-related potential analyses in EEG. Moving beyond this convention, many researchers have begun exploring network-based neurodynamics and coordination between brain regions as a function of behavioral parameters or environmental statistics; however, most approaches average evoked activity across the experimental session to study task-dependent networks. Here, we examined on-going oscillatory activity and use a methodology t…
Université de Montréal（蒙特利尔大学）；Mila – Quebec Artificial Intelligence Institute（米拉-魁北克人工智能研究所）；IMT Atlantique（IMT大西洋国立高等矿业电信学校）；IVADO, Université de Montréal（蒙特利尔大学IVADO研究所）；Hôpital du Sacré-Cœur de Montréal, Santé Québec Nord-de-l’Île-de-Montréal - Universitaire（蒙特利尔圣心医院，魁北克健康部蒙特利尔岛北-大学网络）；Faculty of Medicine, Université de Montréal（蒙特利尔大学医学院）
Understanding Auditory Evoked Brain Signal via Physics-informed Embedding Network with Multi-Task Transformer
School of Mathematics, Southwest Jiaotong University（西南交通大学数学学院）；School of Computer, SWJTU-Leeds Joint School（西南交通大学-利兹学院计算机学院）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；Chengdu University of Technology Oxford Brookes College（成都理工大学牛津布鲁克斯学院）
