Brain decoding, understood as the process of mapping brain activities to the stimuli that generated them, has been an active research area in the last years. In the case of language stimuli, recent studies have shown that it is possible to decode fMRI scans into an embedding of the word a subject is reading. However, such word embeddings are designed for natural language processing tasks rather than for brain decoding. Therefore, they limit our ability to recover the precise stimulus. In this work, we propose to directly classify an fMRI scan, mapping it to the corresponding word within a fixed vocabulary. Unlike existing work, we evaluate on scans from previously unseen subjects. We argue that this is a more realistic setup and we present a model that can decode fMRI data from unseen subjects. Our model achieves 5.22% Top-1 and 13.59% Top-5 accuracy in this challenging task, significantly outperforming all the considered competitive baselines. Furthermore, we use the decoded words to guide language generation with the GPT-2 model. This way, we advance the quest for a system that translates brain activities into coherent text.
Decoding language representations directly from the brain can enable new Brain-Computer Interfaces (BCI) for high bandwidth human-human and human-machine communication. Clinically, such technologies can restore communication in people with neurological conditions affecting their ability to speak. In this study, we propose a novel deep network architecture Brain2Char, for directly decoding text (specifically character sequences) from direct brain recordings (called Electrocorticography, ECoG). Brain2Char framework combines state-of-the-art deep learning modules --- 3D Inception layers for multi…
A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings
Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong Baptist University（香港 Baptist大学）
Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology（麻省理工学院脑与认知科学系）
Language decoding studies have identified word representations which can be used to predict brain activity in response to novel words and sentences (Anderson et al., 2016; Pereira et al., 2018). The unspoken assumption of these studies is that, during processing, linguistic information is transformed into some shared semantic space, and those semantic representations are then used for a variety of linguistic and non-linguistic tasks. We claim that current studies vastly underdetermine the content of these representations, the algorithms which the brain deploys to produce and consume them, and …
EmoMind: Decoding Affective Captions from Human Brain fMRI
Department of Biomedical Engineering（生物医学工程系）；Vanderbilt University（范德比大学）；Research Institute of Electrical Communication（电气通信研究所）；Tohoku University（东北大学）；University of Florida（佛罗里达大学）
Rethinking Cross-Subject Data Splitting for Brain-to-Text Decoding
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（人工智能学院，南京航空航天大学）；The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education（脑机智能技术重点实验室，教育部）
What information from an act of sentence understanding is robustly represented in the human brain? We investigate this question by comparing sentence encoding models on a brain decoding task, where the sentence that an experimental participant has seen must be predicted from the fMRI signal evoked by the sentence. We take a pre-trained BERT architecture as a baseline sentence encoding model and fine-tune it on a variety of natural language understanding (NLU) tasks, asking which lead to improvements in brain-decoding performance. We find that none of the sentence encoding tasks tested yield si…
How the brain captures the meaning of linguistic stimuli across multiple views is still a critical open question in neuroscience. Consider three different views of the concept apartment: (1) picture (WP) presented with the target word label, (2) sentence (S) using the target word, and (3) word cloud (WC) containing the target word along with other semantically related words. Unlike previous efforts, which focus only on single view analysis, in this paper, we study the effectiveness of brain decoding in a zero-shot cross-view learning setup. Further, we propose brain decoding in the novel conte…
Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings
Meta AI；École Normale Supérieure（高等师范学院）；Université PSL（巴黎文理大学）；CNRS（法国国家科学研究中心）；Hospital Foundation Adolphe de Rothschild（阿道夫·德·罗斯柴尔德医院基金会）；Univ. Lille（里尔大学）；Basque Center on Cognition, Brain and Language（巴斯克认知、大脑与语言中心）；Paris Cité University（巴黎城市大学）；Inria（法国国家信息与自动化研究所）；Université Paris-Saclay（巴黎萨克雷大学）；INSERM（法国国家健康与医学研究院）；CEA（法国原子能和替代能源委员会）
Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Department of Computer Science, The University of Manchester（曼彻斯特大学计算机科学系）；Department of Language Science and Technology, Hong Kong Polytechnic University（香港理工大学语言科学与技术系）
Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
Inria Bordeaux（法国国家信息与自动化研究所波尔多分所）；LaBRI, Université de Bordeaux（波尔多大学拉布里实验室）；Université de Bordeaux（波尔多大学）；TU Berlin（柏林工业大学）；National University of Singapore（新加坡国立大学）；Microsoft（微软公司）；International Institute of Information Technology Hyderabad（海得拉巴国际信息技术研究所）；CEA（法国原子能和替代能源委员会）；CNRS（法国国家科学研究中心）
We evaluate 8 different word embedding models on their usefulness for predicting the neural activation patterns associated with concrete nouns. The models we consider include an experiential model, based on crowd-sourced association data, several popular neural and distributional models, and a model that reflects the syntactic context of words (based on dependency parses). Our goal is to assess the cognitive plausibility of these various embedding models, and understand how we can further improve our methods for interpreting brain imaging data. We show that neural word embedding models exhibit…
Decoding Continuous Character-based Language from Non-invasive Brain Recordings
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学脑智能科学与技术研究院）
Language Reconstruction with Brain Predictive Coding from fMRI Data
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education（教育部脑机智能技术重点实验室）；Institute of Trustworthy Embodied AI, Fudan University（复旦大学可信具身智能研究院）
