ASPEN: Spectral-Temporal Fusion for Cross-Subject Brain Decoding
Carnegie Mellon University（卡内基梅隆大学）；Kyung Hee University（Kyung Hee大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；University of Pittsburgh（匹兹堡大学）
Cross-subject generalization in EEG-based brain-computer interfaces (BCIs) remains challenging due to individual variability in neural signals. We investigate whether spectral representations offer more stable features for cross-subject transfer than temporal waveforms. Through correlation analyses across three EEG paradigms (SSVEP, P300, and Motor Imagery), we find that spectral features exhibit consistently higher cross-subject similarity than temporal signals. Motivated by this observation, we introduce ASPEN, a hybrid architecture that combines spectral and temporal feature streams via multiplicative fusion, requiring cross-modal agreement for features to propagate. Experiments across six benchmark datasets reveal that ASPEN is able to dynamically achieve the optimal spectral-temporal balance depending on the paradigm. ASPEN achieves the best unseen-subject accuracy on three of six datasets and competitive performance on others, demonstrating that multiplicative multimodal fusion enables effective cross-subject generalization.
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）
Decoding emotional states from human brain activity plays an important role in brain-computer interfaces. Existing emotion decoding methods still have two main limitations: one is only decoding a single emotion category from a brain activity pattern and the decoded emotion categories are coarse-grained, which is inconsistent with the complex emotional expression of human; the other is ignoring the discrepancy of emotion expression between the left and right hemispheres of human brain. In this paper, we propose a novel multi-view multi-label hybrid model for fine-grained emotion decoding (up to…
Medical Center - University of Freiburg（弗莱堡大学医学中心）；University of Freiburg（弗莱堡大学）
In this manuscript, we investigate deep invertible networks for EEG-based brain signal decoding and find them to generate realistic EEG signals as well as classify novel signals above chance. Further ideas for their regularization towards better decoding accuracies are discussed.
ETH Zurich（苏黎世联邦理工学院）；University of Michigan（密歇根大学）
Brain-machine interfaces (BMIs) are promising for motor rehabilitation and mobility augmentation. High-accuracy and low-power algorithms are required to achieve implantable BMI systems. In this paper, we propose a novel spiking neural network (SNN) decoder for implantable BMI regression tasks. The SNN is trained with enhanced spatio-temporal backpropagation to fully leverage its ability in handling temporal problems. The proposed SNN decoder achieves the same level of correlation coefficient as the state-of-the-art ANN decoder in offline finger velocity decoding tasks, while it requires only 6…
Spiking Neural Network for Intra-cortical Brain Signal Decoding
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；Hubei Key Laboratory of Brain-inspired Intelligent Systems, Huazhong University of Science and Technology（华中科技大学湖北省类脑智能系统重点实验室）；College of Life Science and Technology, Huazhong University of Science and Technology（华中科技大学生命科学与技术学院）
University of Trento（特伦托大学）；Fondazione Bruno Kessler (FBK)（布鲁诺·凯斯勒基金会（FBK））；Centro Interdipartimentale Mente e Cervello (CIMeC)（跨部门心智与大脑中心（CIMeC））
Brain decoding is a popular multivariate approach for hypothesis testing in neuroimaging. It is well known that the brain maps derived from weights of linear classifiers are hard to interpret because of high correlations between predictors, low signal to noise ratios, and the high dimensionality of neuroimaging data. Therefore, improving the interpretability of brain decoding approaches is of primary interest in many neuroimaging studies. Despite extensive studies of this type, at present, there is no formal definition for interpretability of multivariate brain maps. As a consequence, there is…
In this work, we explore the possibility of decoding Imagined Speech brain waves using machine learning techniques. We propose a covariance matrix of Electroencephalogram channels as input features, projection to tangent space of covariance matrices for obtaining vectors from covariance matrices, principal component analysis for dimension reduction of vectors, an artificial feed-forward neural network as a classification model and bootstrap aggregation for creating an ensemble of neural network models. After the classification, two different Finite State Machines are designed that create an in…
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；School of Electrical and Information Engineering, Zhengzhou University（郑州大学电气工程学院）
CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）；Karlsruher Institut fur Technologie（卡尔斯鲁厄理工学院）
Decoding language representations directly from the brain can enable new Brain-Computer Interfaces (BCI) for high bandwidth human-human and human-machine communication. Clinically, such technologies can restore communication in people with neurological conditions affecting their ability to speak. In this study, we propose a novel deep network architecture Brain2Char, for directly decoding text (specifically character sequences) from direct brain recordings (called Electrocorticography, ECoG). Brain2Char framework combines state-of-the-art deep learning modules --- 3D Inception layers for multi…
fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignment
College of Information Science and Engineering, Hunan University（湖南大学信息科学与工程学院）；School of Computer Science and Technology, Nanjing University of Science and Technology（南京理工大学计算机科学与技术学院）；Department of Automation, Tsinghua University（清华大学自动化系）
SEED: Towards More Accurate Semantic Evaluation for Visual Brain Decoding
IPAI, Seoul National University（韩亚国立大学IPAI）；ECE, Seoul National University（韩亚国立大学电子工程系）；Psychology, Seoul National University（韩亚国立大学心理学系）；Brookhaven National Lab（布鲁赫斯国家实验室）；ASRI / INMC / AIIS, Seoul National University（韩亚国立大学ASRI/INMC/AIIS）
Zhejiang University（浙江大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Brain-computer interfaces (BCIs) have enabled prosthetic device control by decoding motor movements from neural activities. Neural signals recorded from cortex exhibit nonstationary property due to abrupt noises and neuroplastic changes in brain activities during motor control. Current state-of-the-art neural signal decoders such as Kalman filter assume fixed relationship between neural activities and motor movements, thus will fail if this assumption is not satisfied. We propose a dynamic ensemble modeling (DyEnsemble) approach that is capable of adapting to changes in neural signals by emplo…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Among the most impressive recent applications of neural decoding is the visual representation decoding, where the category of an object that a subject either sees or imagines is inferred by observing his/her brain activity. Even though there is an increasing interest in the aforementioned visual representation decoding task, there is no extensive study of the effect of using different machine learning models on the decoding accuracy. In this paper we provide an extensive evaluation of several machine learning models, along with different similarity metrics, for the aforementioned task, drawing…
Brain decoding, understood as the process of mapping brain activities to the stimuli that generated them, has been an active research area in the last years. In the case of language stimuli, recent studies have shown that it is possible to decode fMRI scans into an embedding of the word a subject is reading. However, such word embeddings are designed for natural language processing tasks rather than for brain decoding. Therefore, they limit our ability to recover the precise stimulus. In this work, we propose to directly classify an fMRI scan, mapping it to the corresponding word within a fixe…
This literature review will discuss the use of deep learning methods for image reconstruction using fMRI data. More specifically, the quality of image reconstruction will be determined by the choice in decoding and reconstruction architectures. I will show that these structures can struggle with adaptability to various input stimuli due to complicated objects in images. Also, the significance of feature representation will be evaluated. This paper will conclude the use of deep learning within visual decoding and reconstruction is highly optimal when using variations of deep neural networks and…
Middle East Technical University（中东技术大学）；Tohoku University（东北大学）；Koc University（科克大学）
We represent the sequence of fMRI (Functional Magnetic Resonance Imaging) brain volumes recorded during a cognitive stimulus by a graph which consists of a set of local meshes. The corresponding cognitive process, encoded in the brain, is then represented by these meshes each of which is estimated assuming a linear relationship among the voxel time series in a predefined locality. First, we define the concept of locality in two neighborhood systems, namely, the spatial and functional neighborhoods. Then, we construct spatially and functionally local meshes around each voxel, called seed voxel,…
Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings
Meta AI；École Normale Supérieure（高等师范学院）；Université PSL（巴黎文理大学）；CNRS（法国国家科学研究中心）；Hospital Foundation Adolphe de Rothschild（阿道夫·德·罗斯柴尔德医院基金会）；Univ. Lille（里尔大学）；Basque Center on Cognition, Brain and Language（巴斯克认知、大脑与语言中心）；Paris Cité University（巴黎城市大学）；Inria（法国国家信息与自动化研究所）；Université Paris-Saclay（巴黎萨克雷大学）；INSERM（法国国家健康与医学研究院）；CEA（法国原子能和替代能源委员会）
How the brain captures the meaning of linguistic stimuli across multiple views is still a critical open question in neuroscience. Consider three different views of the concept apartment: (1) picture (WP) presented with the target word label, (2) sentence (S) using the target word, and (3) word cloud (WC) containing the target word along with other semantically related words. Unlike previous efforts, which focus only on single view analysis, in this paper, we study the effectiveness of brain decoding in a zero-shot cross-view learning setup. Further, we propose brain decoding in the novel conte…
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions（深港脑科学研究院-深圳基础研究机构）；Shenzhen Technological Research Center for Primate Translational Medicine（深圳灵长类转化医学技术研究中心）；University of Chinese Academy of Sciences（中国科学院大学）
Reading emotions precisely from segments of neural activity is crucial for the development of emotional brain-computer interfaces. Among all neural decoding algorithms, deep learning (DL) holds the potential to become the most promising one, yet progress has been limited in recent years. One possible reason is that the efficacy of DL strongly relies on training samples, yet the neural data used for training are often from non-human primates and mixed with plenty of noise, which in turn mislead the training of DL models. Given it is difficult to accurately determine animals' emotions from human…
Inria（法国国家信息与自动化研究所）；CEA（法国原子能与替代能源委员会）；Univ. Paris Saclay（巴黎萨克雷大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）
Cognitive brain imaging is accumulating datasets about the neural substrate of many different mental processes. Yet, most studies are based on few subjects and have low statistical power. Analyzing data across studies could bring more statistical power; yet the current brain-imaging analytic framework cannot be used at scale as it requires casting all cognitive tasks in a unified theoretical framework. We introduce a new methodology to analyze brain responses across tasks without a joint model of the psychological processes. The method boosts statistical power in small studies with specific co…
Few-shot learning addresses problems for which a limited number of training examples are available. So far, the field has been mostly driven by applications in computer vision. Here, we are interested in adapting recently introduced few-shot methods to solve problems dealing with neuroimaging data, a promising application field. To this end, we create a neuroimaging benchmark dataset for few-shot learning and compare multiple learning paradigms, including meta-learning, as well as various backbone networks. Our experiments show that few-shot methods are able to efficiently decode brain signals…
Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
Inria Bordeaux（法国国家信息与自动化研究所波尔多分所）；LaBRI, Université de Bordeaux（波尔多大学拉布里实验室）；Université de Bordeaux（波尔多大学）；TU Berlin（柏林工业大学）；National University of Singapore（新加坡国立大学）；Microsoft（微软公司）；International Institute of Information Technology Hyderabad（海得拉巴国际信息技术研究所）；CEA（法国原子能和替代能源委员会）；CNRS（法国国家科学研究中心）
Tailoring deep learning for real-time brain-computer interfaces: From offline models to calibration-free online decoding
Institute of Signal Processing and System Theory, University of Stuttgart（信号处理与系统理论研究所，斯图加特大学）；Clinical Neurotechnology Laboratory, Department of Psychiatry and Neurosciences, Charité Campus Mitte (CCM), Charité - Universitätsmedizin Berlin（临床神经技术实验室，精神病学与神经科学系，Charité-柏林大学医学院）
EmoMind: Decoding Affective Captions from Human Brain fMRI
Department of Biomedical Engineering（生物医学工程系）；Vanderbilt University（范德比大学）；Research Institute of Electrical Communication（电气通信研究所）；Tohoku University（东北大学）；University of Florida（佛罗里达大学）
Reconstructing perceived images from human brain activity monitored by functional magnetic resonance imaging (fMRI) is hard, especially for natural images. Existing methods often result in blurry and unintelligible reconstructions with low fidelity. In this study, we present a novel approach for enhanced image reconstruction, in which existing methods for object decoding and image reconstruction are merged together. This is achieved by conditioning the reconstructed image to its decoded image category using a class-conditional generative adversarial network and neural style transfer. The resul…
Fast Whole-Brain, Geometry-Aware Functional Alignment for Cross-Subject Decoding
Université Paris-Saclay（巴黎-萨克雷大学）；Inria（法国国家信息与自动化技术研究院）；CEA（法国原子能委员会）；Cognitive Neuroimaging Unit（认知神经成像单位）；Institut National de la Santé et de la Recherche Médicale（法国国家卫生与医学研究院）；Commissariat à l’Energie Atomique et aux énergies alternatives（法国原子能委员会及替代能源委员会）；NeuroSpin center（神经科学中心）；Institut de neuromodulation（神经调制研究所）；Université Paris Cité（巴黎-城市大学）
University of Hong Kong（香港大学）；Shenzhen Loop Area Institute（深圳河套学院）；Chinese University of Hong Kong（香港中文大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Harvard University（哈佛大学）；Columbia University（哥伦比亚大学）；University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）
Technische Universität Berlin（柏林工业大学）；Max Planck Institute for Human Development（马克斯·普朗克人类发展研究所）；Stanford University（斯坦福大学）；Max Planck UCL Centre for Computational Psychiatry and Ageing Research（马克斯·普朗克伦敦大学学院计算精神病学与老龄化研究中心）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；Korea University（高丽大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Research in many fields has shown that transfer learning (TL) is well-suited to improve the performance of deep learning (DL) models in datasets with small numbers of samples. This empirical success has triggered interest in the application of TL to cognitive decoding analyses with functional neuroimaging data. Here, we systematically evaluate TL for the application of DL models to the decoding of cognitive states (e.g., viewing images of faces or houses) from whole-brain functional Magnetic Resonance Imaging (fMRI) data. We first pre-train two DL architectures on a large, public fMRI dataset …
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）
A universal unanswered question in neuroscience and machine learning is whether computers can decode the patterns of the human brain. Multi-Voxels Pattern Analysis (MVPA) is a critical tool for addressing this question. However, there are two challenges in the previous MVPA methods, which include decreasing sparsity and noises in the extracted features and increasing the performance of prediction. In overcoming mentioned challenges, this paper proposes Anatomical Pattern Analysis (APA) for decoding visual stimuli in the human brain. This framework develops a novel anatomical feature extraction…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Multivariate Pattern (MVP) classification holds enormous potential for decoding visual stimuli in the human brain by employing task-based fMRI data sets. There is a wide range of challenges in the MVP techniques, i.e. decreasing noise and sparsity, defining effective regions of interest (ROIs), visualizing results, and the cost of brain studies. In overcoming these challenges, this paper proposes a novel model of neural representation, which can automatically detect the active regions for each visual stimulus and then utilize these anatomical regions for visualizing and analyzing the functiona…
