Toward a realistic model of speech processing in the brain with self-supervised learning
Ecole Normale Supérieure, PSL University(巴黎高等师范学院,巴黎文理研究大学);Université de Paris cité(巴黎城市大学);Université Paris-Saclay(巴黎萨克雷大学);Inria(法国国家信息与自动化研究所);CEA(法国原子能和替代能源委员会);University of Toronto(多伦多大学);INSERM(法国国家健康与医学研究院);Meta AI
Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King
Several deep neural networks have recently been shown to generate activations similar to those of the brain in response to the same input. These algorithms, however, remain largely implausible: they require (1) extraordinarily large amounts of data, (2) unobtainable supervised labels, (3) textual rather than raw sensory input, and / or (4) implausibly large memory (e.g. thousands of contextual words). These elements highlight the need to identify algorithms that, under these limitations, would suffice to account for both behavioral and brain responses. Focusing on the issue of speech processing, we here hypothesize that self-supervised algorithms trained on the raw waveform constitute a promising candidate. Specifically, we compare a recent self-supervised architecture, Wav2Vec 2.0, to the brain activity of 412 English, French, and Mandarin individuals recorded with functional Magnetic Resonance Imaging (fMRI), while they listened to ~1h of audio books. Our results are four-fold. First, we show that this algorithm learns brain-like representations with as little as 600 hours of unlabelled speech -- a quantity comparable to what infants can be exposed to during language acquisition. Second, its functional hierarchy aligns with the cortical hierarchy of speech processing. Third, different training regimes reveal a functional specialization akin to the cortex: Wav2Vec 2.0 learns sound-generic, speech-specific and language-specific representations similar to those of the prefrontal and temporal cortices. Fourth, we confirm the similarity of this specialization with the behavior of 386 additional participants. These elements, resulting from the largest neuroimaging benchmark to date, show how self-supervised learning can account for a rich organization of speech processing in the brain, and thus delineate a path to identify the laws of language acquisition which shape the human brain.
MEBM-Speech: Multi-scale Enhanced BrainMagic for Robust MEG Speech Detection
Speech and Hearing Research Center, School of Intelligence Science and Technology(语音与听力研究中心,智能科学与技术学院);Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies(BioMed-X研究中心,先进跨学科研究学院)
Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech
Facebook AI Research(Facebook AI研究院);Université de Paris(巴黎大学);CoML, ENS/CNRS/EHESS/INRIA/PSL University(计算语言学实验室,巴黎高等师范学院/法国国家科学研究中心/法国社会科学高等研究院/法国国家信息与自动化研究所/巴黎文理研究大学);CRI, Département Frontières du Vivant et de l’Apprendre, IIFR, Université de Paris(认知研究所,生命与学习前沿系,法国医学研究院,巴黎大学);École normale supérieure, PSL University, CNRS(巴黎高等师范学院,巴黎文理研究大学,法国国家科学研究中心)
Towards unified brain-to-text decoding across speech production and perception
Computer Science and Technology(计算机科学与技术);Zhejiang University(浙江大学);Shanghai Institute of Microsystem and Information Technology(上海微系统与信息工程研究所);Chinese Academy of Sciences(中国科学院);Department of Neurosurgery(神经外科部门);Huashan Hospital, Fudan University(复旦大学华山医院)
本文提出一种统一的脑-句子解码框架,用于 Mandarin Chinese 的语音生成与感知,通过单字符数据训练实现句子级解码,并通过多模态神经动态比较提升解码性能。
Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals
Peking University(北京大学);Chinese Institute for Brain Research(中国脑科学研究院);Beijing Normal University(北京师范大学);Shanghai Jiao Tong University(上海交通大学);Capital Medical University(首都医科大学);Xuanwu Hospital(宣武医院)
Longitudinal Acoustic Speech Tracking Following Pediatric Traumatic Brain Injury
Stanford University(斯坦福大学);MIT Lincoln Laboratory(麻省理工学院林肯实验室);Worcester Polytechnic Institute(伍斯特理工学院);Mass General Hospital(麻省总医院);University of Texas at Dallas(德克萨斯大学达拉斯分校)