Joint processing of linguistic properties in brains and language models
Inria Bordeaux（法国国家信息与自动化研究所波尔多分部）；MPI for Software Systems（马克斯·普朗克软件系统研究所）；Microsoft（微软）
Subba Reddy Oota, Manish Gupta, Mariya Toneva
备注：
22 pages, 12 figures, To be published in the proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), New Orleans, USA
Language models have been shown to be very effective in predicting brain recordings of subjects experiencing complex language stimuli. For a deeper understanding of this alignment, it is important to understand the correspondence between the detailed processing of linguistic information by the human brain versus language models. We investigate this correspondence via a direct approach, in which we eliminate information related to specific linguistic properties in the language model representations and observe how this intervention affects the alignment with fMRI brain recordings obtained while participants listened to a story. We investigate a range of linguistic properties (surface, syntactic, and semantic) and find that the elimination of each one results in a significant decrease in brain alignment. Specifically, we find that syntactic properties (i.e. Top Constituents and Tree Depth) have the largest effect on the trend of brain alignment across model layers. These findings provide clear evidence for the role of specific linguistic information in the alignment between brain and language models, and open new avenues for mapping the joint information processing in both systems. We make the code publicly available [https://github.com/subbareddy248/linguistic-properties-brain-alignment].
Linguistics and Human Brain: A Perspective of Computational Neuroscience
Institute of AI and Robotics, College of Intelligent Robotics and Advanced Manufacturing, Fudan University（人工智能与机器人研究院，智能机器人与先进制造学院，复旦大学）；Department of Language Science and Technology, The Hong Kong Polytechnic University（语言科学与技术系，香港理工大学）；Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）
University of Copenhagen（哥本哈根大学）；Carnegie Mellon University（卡内基梅隆大学）
Neuroscientists evaluate deep neural networks for natural language processing as possible candidate models for how language is processed in the brain. These models are often trained without explicit linguistic supervision, but have been shown to learn some linguistic structure in the absence of such supervision (Manning et al., 2020), potentially questioning the relevance of symbolic linguistic theories in modeling such cognitive processes (Warstadt and Bowman, 2020). We evaluate across two fMRI datasets whether language models align better with brain recordings, if their attention is biased b…
As the name implies, contextualized representations of language are typically motivated by their ability to encode context. Which aspects of context are captured by such representations? We introduce an approach to address this question using Representational Similarity Analysis (RSA). As case studies, we investigate the degree to which a verb embedding encodes the verb's subject, a pronoun embedding encodes the pronoun's antecedent, and a full-sentence representation encodes the sentence's head word (as determined by a dependency parse). In all cases, we show that BERT's contextualized embedd…
Enabling effective brain-computer interfaces requires understanding how the human brain encodes stimuli across modalities such as visual, language (or text), etc. Brain encoding aims at constructing fMRI brain activity given a stimulus. There exists a plethora of neural encoding models which study brain encoding for single mode stimuli: visual (pretrained CNNs) or text (pretrained language models). Few recent papers have also obtained separate visual and text representation models and performed late-fusion using simple heuristics. However, previous work has failed to explore: (a) the effective…
Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology（麻省理工学院脑与认知科学系）
Language decoding studies have identified word representations which can be used to predict brain activity in response to novel words and sentences (Anderson et al., 2016; Pereira et al., 2018). The unspoken assumption of these studies is that, during processing, linguistic information is transformed into some shared semantic space, and those semantic representations are then used for a variety of linguistic and non-linguistic tasks. We claim that current studies vastly underdetermine the content of these representations, the algorithms which the brain deploys to produce and consume them, and …
Left-right asymmetry in predicting brain activity from LLMs' representations emerges with their formal linguistic competence
Centre d’Analyse et de Mathématique Sociales CNRS, EHESS, Paris, France（分析与数学社会中心 CNRS，EHESS，巴黎，法国）；Cognitive Neuroimaging Unit CNRS, INSERM, CEA, Neurospin Center, 91191 Gif-sur-Yvette, France（认知神经成像单元 CNRS，INSERM，CEA，Neurospin中心，法国91191 Gif-sur-Yvette）
Institute for Logic, Language and Computation（逻辑、语言与计算研究所）；University of Amsterdam（阿姆斯特丹大学）
In this paper, we define and apply representational stability analysis (ReStA), an intuitive way of analyzing neural language models. ReStA is a variant of the popular representational similarity analysis (RSA) in cognitive neuroscience. While RSA can be used to compare representations in models, model components, and human brains, ReStA compares instances of the same model, while systematically varying single model parameter. Using ReStA, we study four recent and successful neural language models, and evaluate how sensitive their internal representations are to the amount of prior context. Us…
Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement
Zuckerman Mind Brain Behavior Institute, Columbia University（扎克曼脑行为研究所，哥伦比亚大学）；Department of Electrical Engineering, Columbia University（电气工程系，哥伦比亚大学）；Department of Computer Science, Columbia University（计算机科学系，哥伦比亚大学）
Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment
Department of Linguistics and Modern Languages, The Chinese University of Hong Kong（语言学与现代语言系，香港中文大学）；Brain and Mind Institute, The Chinese University of Hong Kong（脑与心智研究所，香港中文大学）
The dispute of how the human brain represents conceptual knowledge has been argued in many scientific fields. Brain imaging studies have shown that the spatial patterns of neural activation in the brain are correlated with thinking about different semantic categories of words (for example, tools, animals, and buildings) or when viewing the related pictures. In this paper, we present a computational model that learns to predict the neural activation captured in functional magnetic resonance imaging (fMRI) data of test words. Unlike the models with hand-crafted features that have been used in th…
Inria（法国国家信息与自动化研究所）；Facebook AI Research（脸书人工智能研究院）；École normale supérieure（巴黎高等师范学院）；PSL University（巴黎文理研究大学）；CNRS（法国国家科学研究中心）
The activations of language transformers like GPT-2 have been shown to linearly map onto brain activity during speech comprehension. However, the nature of these activations remains largely unknown and presumably conflate distinct linguistic classes. Here, we propose a taxonomy to factorize the high-dimensional activations of language models into four combinatorial classes: lexical, compositional, syntactic, and semantic representations. We then introduce a statistical method to decompose, through the lens of GPT-2's activations, the brain activity of 345 subjects recorded with functional magn…
Multilingual sentence encoders are widely used to transfer NLP models across languages. The success of this transfer is, however, dependent on the model's ability to encode the patterns of cross-lingual similarity and variation. Yet, little is known as to how these models are able to do this. We propose a simple method to study how relationships between languages are encoded in two state-of-the-art multilingual models (i.e. M-BERT and XLM-R). The results provide insight into their information sharing mechanisms and suggest that linguistic properties are encoded jointly across typologically-sim…
Instituto Nacional de Astrofísica, Óptica y Electrónica（国家天体物理、光学与电子研究所）
A Brain Computer Interface based on imagined words can decode the word a subject is thinking on through brain signals to control an external device. In order to build a fully asynchronous Brain Computer Interface based on imagined words in electroencephalogram signals as source, we need to solve the problem of detecting the onset of the imagined words. Although there has been some research in this field, the problem has not been fully solved. In this paper we present an approach to solve this problem by using values from statistics, information theory and chaos theory as features to correctly …
A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings
Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong Baptist University（香港 Baptist大学）
University of Oxford（牛津大学）；Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所）；Radboud University（拉德堡德大学）
Recent artificial neural networks that process natural language achieve unprecedented performance in tasks requiring sentence-level understanding. As such, they could be interesting models of the integration of linguistic information in the human brain. We review works that compare these artificial language models with human brain activity and we assess the extent to which this approach has improved our understanding of the neural processes involved in natural language comprehension. Two main results emerge. First, the neural representation of word meaning aligns with the context-dependent, de…
Northwestern Polytechnical University（西北工业大学）；University of Georgia（佐治亚大学）；Athens Academy（雅典学院）
Linking computational natural language processing (NLP) models and neural responses to language in the human brain on the one hand facilitates the effort towards disentangling the neural representations underpinning language perception, on the other hand provides neurolinguistics evidence to evaluate and improve NLP models. Mappings of an NLP model's representations of and the brain activities evoked by linguistic input are typically deployed to reveal this symbiosis. However, two critical problems limit its advancement: 1) The model's representations (artificial neurons, ANs) rely on layer-le…
In this treatment a text is considered to be a series of word impulses which are read at a constant rate. The brain then assembles these units of information into higher units of meaning. A classical systems approach is used to model an initial part of this assembly process. The concepts of linguistic system response, information energy, and ordering energy are defined and analyzed. Finally, as a demonstration, information energy is used to estimate the publication dates of a series of texts and the similarity of a set of texts.
Decoding Predictive Inference in Visual Language Processing via Spatiotemporal Neural Coherence
Department of Physics（物理系）；Carnegie-Mellon University（卡内基梅隆大学）；Department of Linguistics（语言学系）；University of Salzburg（萨尔茨堡大学）；Purdue University（普渡大学）；University of Alabama（阿拉巴马大学）；Department of Communicative Disorders（沟通障碍系）
Debate and deliberation play essential roles in politics and government, but most models presume that debates are won mainly via superior style or agenda control. Ideally, however, debates would be won on the merits, as a function of which side has the stronger arguments. We propose a predictive model of debate that estimates the effects of linguistic features and the latent persuasive strengths of different topics, as well as the interactions between the two. Using a dataset of 118 Oxford-style debates, our model's combination of content (as latent topics) and style (as linguistic features) a…
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
The recent emergence of joint CTC-Attention model shows significant improvement in automatic speech recognition (ASR). The improvement largely lies in the modeling of linguistic information by decoder. The decoder joint-optimized with an acoustic encoder renders the language model from ground-truth sequences in an auto-regressive manner during training. However, the training corpus of the decoder is limited to the speech transcriptions, which is far less than the corpus needed to train an acceptable language model. This leads to poor robustness of decoder. To alleviate this problem, we propose…
Lost in Translation: The Algorithmic Gap Between LMs and the Brain
CHU Sainte Justine Research Center（圣贾斯汀医院研究中心）；Université de Montréal（蒙特利尔大学）；Mila（米拉研究所）；Ernst Strüngmann Institute for Neuroscience in Cooperation with Max Planck Society（恩斯特·施特龙格曼神经科学研究所（与马克斯·普朗克学会合作））
Alzheimer's disease (AD) is an irreversible and progressive brain disease that can be stopped or slowed down with medical treatment. Language changes serve as a sign that a patient's cognitive functions have been impacted, potentially leading to early diagnosis. In this work, we use NLP techniques to classify and analyze the linguistic characteristics of AD patients using the DementiaBank dataset. We apply three neural models based on CNNs, LSTM-RNNs, and their combination, to distinguish between language samples from AD and control patients. We achieve a new independent benchmark accuracy for…
BITS Pilani, K.K. Birla Goa Campus（Birla技术与科学学院皮拉尼校区 K.K. Birla果阿校区）；IEMN Université de Lille（里尔大学微电子与纳米技术研究所）
Alzheimers disease is a fatal progressive brain disorder that worsens with time. It is high time we have inexpensive and quick clinical diagnostic techniques for early detection and care. In previous studies, various Machine Learning techniques and Pre-trained Deep Learning models have been used in conjunction with the extraction of various acoustic and linguistic features. Our study focuses on three models for the classification task in the ADReSS (The Alzheimers Dementia Recognition through Spontaneous Speech) 2021 Challenge. We use the well-balanced dataset provided by the ADReSS Challenge …
Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Department of Computer Science, The University of Manchester（曼彻斯特大学计算机科学系）；Department of Language Science and Technology, Hong Kong Polytechnic University（香港理工大学语言科学与技术系）
Decoding Continuous Character-based Language from Non-invasive Brain Recordings
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学脑智能科学与技术研究院）
Convergent Representations of Linguistic Constructions in Human and Artificial Neural Systems
Department of English and American Studies, University Erlangen-Nuremberg（英语与美国研究系，埃尔朗根-纽伦堡大学）；Pattern Recognition Lab, University Erlangen-Nuremberg（模式识别实验室，埃尔朗根-纽伦堡大学）；Neuromodulation and Neuroprosthetics, University Hospital Mannheim, University Heidelberg（神经调控与神经假体，曼海姆大学医院，海德堡大学）；BGU Ludwigshafen, Germany（吕贝克大学吕贝克分校，德国）；Neuroscience Lab, University Hospital Erlangen（神经科学实验室，埃尔朗根大学医院）
Understanding images and text together is an important aspect of cognition and building advanced Artificial Intelligence (AI) systems. As a community, we have achieved good benchmarks over language and vision domains separately, however joint reasoning is still a challenge for state-of-the-art computer vision and natural language processing (NLP) systems. We propose a novel task to derive joint inference about a given image-text modality and compile the Visuo-Linguistic Question Answering (VLQA) challenge corpus in a question answering setting. Each dataset item consists of an image and a read…
