arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2206.01685q-bio.NCcs.AIcs.CL
收藏

通过自监督学习建立大脑语音处理的现实模型

Toward a realistic model of speech processing in the brain with self-supervised learning

Ecole Normale Supérieure, PSL University(巴黎高等师范学院,巴黎文理研究大学);Université de Paris cité(巴黎城市大学);Université Paris-Saclay(巴黎萨克雷大学);Inria(法国国家信息与自动化研究所);CEA(法国原子能和替代能源委员会);University of Toronto(多伦多大学);INSERM(法国国家健康与医学研究院);Meta AI

Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King

摘要:

最近,几种深度神经网络被证明可以产生类似于大脑的激活,以响应相同的输入。然而,这些算法在很大程度上仍然令人难以置信:它们需要(1)非常大量的数据,(2)无法获得的监督标签,(3)文本而不是原始的感官输入,和/或(4)难以置信的大内存(例如数千个上下文单词)。这些因素强调了识别算法的必要性,在这些限制下,这些算法足以解释行为和大脑反应。专注于语音处理的问题,我们在这里假设,在原始波形上训练的自监督算法构成了一个有前途的候选人。具体来说,我们将最近的自我监督架构Wav 2 Vec 2.0与412名使用功能性磁共振成像(fMRI)记录的英语,法语和普通话个体的大脑活动进行了比较,同时他们听了约1小时的有声读物。我们的成果有四个方面。首先,我们证明了这种算法只需600小时的未标记语音就可以学习类似大脑的表示-这个数量与婴儿在语言习得过程中可以接触到的数量相当。其次,它的功能层次与语音处理的皮层层次相一致。第三,不同的训练机制揭示了类似于皮层的功能专业化:Wav 2 Vec 2.0学习与前额叶和颞叶皮层相似的声音通用,语音特定和语言特定的表征。第四,我们证实了这种专业化的相似性与386个额外的参与者的行为。这些元素来自迄今为止最大的神经成像基准,展示了自我监督学习如何解释大脑中丰富的语音处理组织,从而描绘了一条识别塑造人类大脑的语言习得规律的路径。

英文摘要:

Several deep neural networks have recently been shown to generate activations similar to those of the brain in response to the same input. These algorithms, however, remain largely implausible: they require (1) extraordinarily large amounts of data, (2) unobtainable supervised labels, (3) textual rather than raw sensory input, and / or (4) implausibly large memory (e.g. thousands of contextual words). These elements highlight the need to identify algorithms that, under these limitations, would suffice to account for both behavioral and brain responses. Focusing on the issue of speech processing, we here hypothesize that self-supervised algorithms trained on the raw waveform constitute a promising candidate. Specifically, we compare a recent self-supervised architecture, Wav2Vec 2.0, to the brain activity of 412 English, French, and Mandarin individuals recorded with functional Magnetic Resonance Imaging (fMRI), while they listened to ~1h of audio books. Our results are four-fold. First, we show that this algorithm learns brain-like representations with as little as 600 hours of unlabelled speech -- a quantity comparable to what infants can be exposed to during language acquisition. Second, its functional hierarchy aligns with the cortical hierarchy of speech processing. Third, different training regimes reveal a functional specialization akin to the cortex: Wav2Vec 2.0 learns sound-generic, speech-specific and language-specific representations similar to those of the prefrontal and temporal cortices. Fourth, we confirm the similarity of this specialization with the behavior of 386 additional participants. These elements, resulting from the largest neuroimaging benchmark to date, show how self-supervised learning can account for a rich organization of speech processing in the brain, and thus delineate a path to identify the laws of language acquisition which shape the human brain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑