We create a reusable Transformer, BrainBERT, for intracranial recordings bringing modern representation learning approaches to neuroscience. Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, i.e., decoding neural data, with higher accuracy and with much less data by being pretrained in an unsupervised manner on a large corpus of unannotated neural recordings. Our approach generalizes to new subjects with electrodes in new positions and to unrelated tasks showing that the representations robustly disentangle the neural signal. Just like in NLP where one can study language by investigating what a language model learns, this approach opens the door to investigating the brain by what a model of the brain learns. As a first step along this path, we demonstrate a new analysis of the intrinsic dimensionality of the computations in different areas of the brain. To construct these representations, we combine a technique for producing super-resolution spectrograms of neural data with an approach designed for generating contextual representations of audio by masking. In the future, far more concepts will be decodable from neural recordings by using representation learning, potentially unlocking the brain like language models unlocked language.
Identifying stimulus-driven neural activity patterns is critical for studying the neural basis of cognition. This can be particularly challenging in intracranial datasets, where electrode locations typically vary across patients. This chapter first presents an overview of the major challenges to identifying stimulus-driven neural activity patterns in the general case. Next, we will review several modality-specific considerations and approaches, along with a discussion of several issues that are particular to intracranial recordings. Against this backdrop, we will consider a variety of within-s…
Nonlinear Dynamical Modeling of Human Intracranial Brain Activity with Flexible Inference
Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California（明希学院电气与计算机工程系，维特比工程学院，南加州大学）；Keck School of Medicine, University of Southern California（凯克医学院，南加州大学）
ETH Zurich（苏黎世联邦理工学院）；Maastricht University（马斯特里赫特大学）；University of Bremen（不来梅大学）；Maastricht University Medical Center（马斯特里赫特大学医学中心）；Kempenhaeghe（坎彭海格癫痫中心）
Speech Neuroprostheses have the potential to enable communication for people with dysarthria or anarthria. Recent advances have demonstrated high-quality text decoding and speech synthesis from electrocorticographic grids placed on the cortical surface. Here, we investigate a less invasive measurement modality in three participants, namely stereotactic EEG (sEEG) that provides sparse sampling from multiple brain regions, including subcortical regions. To evaluate whether sEEG can also be used to synthesize audio from neural recordings, we employ a recurrent encoder-decoder model based on moder…
Functional embeddings enable Aggregation of multi-area SEEG recordings over subjects and sessions
Department of Biomedical Engineering, University of California Irvine（生物医学工程系，加州大学尔湾分校）；Department of Electrical Engineering, University of California Irvine（电气工程系，加州大学尔湾分校）；Department of Neurology, Rady Children’s Health（神经病学系，Rady儿童健康中心）
Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings
Meta AI；École Normale Supérieure（高等师范学院）；Université PSL（巴黎文理大学）；CNRS（法国国家科学研究中心）；Hospital Foundation Adolphe de Rothschild（阿道夫·德·罗斯柴尔德医院基金会）；Univ. Lille（里尔大学）；Basque Center on Cognition, Brain and Language（巴斯克认知、大脑与语言中心）；Paris Cité University（巴黎城市大学）；Inria（法国国家信息与自动化研究所）；Université Paris-Saclay（巴黎萨克雷大学）；INSERM（法国国家健康与医学研究院）；CEA（法国原子能和替代能源委员会）
Caltech（加州理工学院）；UCSF（加州大学旧金山分校）；U of Washington（华盛顿大学）
Neuroscientists and neuroengineers have long relied on multielectrode neural recordings to study the brain. However, in a typical experiment, many factors corrupt neural recordings from individual electrodes, including electrical noise, movement artifacts, and faulty manufacturing. Currently, common practice is to discard these corrupted recordings, reducing already limited data that is difficult to collect. To address this challenge, we propose Deep Neural Imputation (DNI), a framework to recover missing values from electrodes by learning from data collected across spatial locations, days, an…
Epilepsy Center Frankfurt（法兰克福癫痫中心）；Rhine-Main Neurocenter（莱茵-美因神经中心）
Epilepsy is a common neurological disorder characterized by recurrent seizures accompanied by excessive synchronous brain activity. The process of structural and functional brain alterations leading to increased seizure susceptibility and eventually spontaneous seizures is called epileptogenesis (EPG) and can span months or even years. Detecting and monitoring the progression of EPG could allow for targeted early interventions that could slow down disease progression or even halt its development. Here, we propose an approach for staging EPG using deep neural networks and identify potential ele…
Improving Speech Decoding from ECoG with Self-Supervised Pretraining
Institute of Neuroinformatics（神经信息学研究所）；University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）；School of Electrical and Computer Engineering（电气与计算机工程学院）；Purdue University（普渡大学）
Scientific Machine Learning of Chaotic Systems Learns Reduced-Order Equations for Neural Populations
Department of Biomedical Engineering and Laufer Center for Physical and Quantitative Biology, State University of New York at Stony Brook, NY, USA（生物医学工程系和物理与定量生物学拉夫中心，石溪大学纽约州立大学 Stony Brook 分校）；Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School, MA, USA（Athinoula A. Martinos 生物医学成像中心，麻省总医院和哈佛医学院）；Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, MA, USA（计算机科学与人工智能实验室，麻省理工学院）；Picower Institute for Learning and Memory, Massachusetts Institute of Technology, MA, USA（记忆学习研究所，麻省理工学院）；Psychological and Brain Sciences, Dartmouth College, NH, USA（心理学与脑科学系，达特茅斯学院）；Santa Fe Institute, NM, USA（圣菲研究所）
Online decoding of rat self-paced locomotion speed from EEG using recurrent neural networks
Schmid College of Science and Technology, Chapman University（施密特科学与技术学院，查普曼大学）；LUCID-AI, Chapman University（LUCID-AI，查普曼大学）；Helsinki Institute of Life Science, University of Helsinki（赫尔辛基生命科学研究所，赫尔辛基大学）；Neuroscience Center, University of Helsinki（神经科学中心，赫尔辛基大学）；Faculty of Pharmacy, University of Helsinki（药学院，赫尔辛基大学）；Crean College of Health and Behavioral Sciences, Chapman University（克里安健康与行为科学学院，查普曼大学）；Department of Biology and Bioengineering, California Institute of Technology（生物学与生物工程系，加州理工学院）；Anderson School of Management, University of California Los Angeles（安德森管理学院，加州大学洛杉矶分校）
