Brain Signals, such as Electroencephalography (EEG), and human languages have been widely explored independently for many downstream tasks, however, the connection between them has not been well explored. In this study, we explore the relationship and dependency between EEG and language. To study at the representation level, we introduced \textbf{MTAM}, a \textbf{M}ultimodal \textbf{T}ransformer \textbf{A}lignment \textbf{M}odel, to observe coordinated representations between the two modalities. We used various relationship alignment-seeking techniques, such as Canonical Correlation Analysis and Wasserstein Distance, as loss functions to transfigure features. On downstream applications, sentiment analysis and relation detection, we achieved new state-of-the-art results on two datasets, ZuCo and K-EmoCon. Our method achieved an F1-score improvement of 1.7% on K-EmoCon and 9.3% on Zuco datasets for sentiment analysis, and 7.4% on ZuCo for relation detection. In addition, we provide interpretations of the performance improvement: (1) feature distribution shows the effectiveness of the alignment module for discovering and encoding the relationship between EEG and language; (2) alignment weights show the influence of different language semantics as well as EEG frequency features; (3) brain topographical maps provide an intuitive demonstration of the connectivity in the brain regions. Our code is available at \url{https://github.com/Jason-Qiu/EEG_Language_Alignment}.
Beyond representational alignment with brain-guided language models for robust reasoning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（北京大学通用人工智能国家重点实验室、智能科学与技术学院）；Department of Psychological and Cognitive Sciences, Tsinghua University（清华大学心理与认知科学系）；Microsoft Research Asia（微软亚洲研究院）
Large Language Models Align with the Human Brain during Creative Thinking
EPFL（瑞士联邦理工学院洛桑）；Università della Svizzera italiana (USI)（意大利语区大学（瑞士））；Wesleyan University（卫斯理大学）；Paris Brain Institute (ICM)（巴黎大脑研究所）；Pennsylvania State University（宾夕法尼亚州立大学）
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）
Neural encoding and decoding, which aim to characterize the relationship between stimuli and brain activities, have emerged as an important area in cognitive neuroscience. Traditional encoding models, which focus on feature extraction and mapping, consider the brain as an input-output mapper without inner states. In this work, inspired by the fact that human brain acts like a state machine, we proposed a novel encoding framework that combines information from both the external world and the inner state to predict brain activity. The framework comprises two parts: forward encoding model that de…
Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
Technische Universität Berlin（柏林技术大学）；IIIT Hyderabad（海得拉巴国家理工学院）；Univ of Maryland（马里兰大学）；Rice Univ（Rice 大学）；Univ of Wisconsin - Madison（威斯康星大学麦迪逊分校）；Spector Inc（Spector 公司）；Microsoft（微软公司）
Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；Singapore Management University（新加坡管理大学）
Building systems that achieve a deeper understanding of language is one of the central goals of natural language processing (NLP). Towards this goal, recent works have begun to train language models on narrative datasets which require extracting the most critical information by integrating across long contexts. However, it is still an open question whether these models are learning a deeper understanding of the text, or if the models are simply learning a heuristic to complete the task. This work investigates this further by turning to the one language processing system that truly understands …
Fast Whole-Brain, Geometry-Aware Functional Alignment for Cross-Subject Decoding
Université Paris-Saclay（巴黎-萨克雷大学）；Inria（法国国家信息与自动化技术研究院）；CEA（法国原子能委员会）；Cognitive Neuroimaging Unit（认知神经成像单位）；Institut National de la Santé et de la Recherche Médicale（法国国家卫生与医学研究院）；Commissariat à l’Energie Atomique et aux énergies alternatives（法国原子能委员会及替代能源委员会）；NeuroSpin center（神经科学中心）；Institut de neuromodulation（神经调制研究所）；Université Paris Cité（巴黎-城市大学）
Explanations of Large Language Models Explain Language Representations in the Brain
Iran University of Science and Technology（伊朗科学技术大学）；University of Tehran（德黑兰大学）；Institute for Research in Fundamental Sciences（基础科学研究所）；Khatam University（哈塔姆大学）
Achieving More Human Brain-Like Vision via Human EEG Representational Alignment
Department of Psychology, The Ohio State University（心理学系，俄亥俄州立大学）；McGovern Institute for Brain Research, Massachusetts Institute of Technology（脑科学研究所，麻省理工学院）；Department of Neuroscience, The University of Texas at Dallas（神经科学系，德克萨斯大学达拉斯分校）
Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
Inria Bordeaux（法国国家信息与自动化研究所波尔多分所）；LaBRI, Université de Bordeaux（波尔多大学拉布里实验室）；Université de Bordeaux（波尔多大学）；TU Berlin（柏林工业大学）；National University of Singapore（新加坡国立大学）；Microsoft（微软公司）；International Institute of Information Technology Hyderabad（海得拉巴国际信息技术研究所）；CEA（法国原子能和替代能源委员会）；CNRS（法国国家科学研究中心）
Istanbul Technical University（伊斯坦布尔理工大学）；University of Sousse（苏塞大学）
Accurate and automated super-resolution image synthesis is highly desired since it has the great potential to circumvent the need for acquiring high-cost medical scans and a time-consuming preprocessing pipeline of neuroimaging data. However, existing deep learning frameworks are solely designed to predict high-resolution (HR) image from a low-resolution (LR) one, which limits their generalization ability to brain graphs (i.e., connectomes). A small body of works has focused on superresolving brain graphs where the goal is to predict a HR graph from a single LR graph. Although promising, exist…
Huazhong University of Science and Technology（华中科技大学）
Objective: This paper targets a major challenge in developing practical EEG-based brain-computer interfaces (BCIs): how to cope with individual differences so that better learning performance can be obtained for a new subject, with minimum or even no subject-specific data? Methods: We propose a novel approach to align EEG trials from different subjects in the Euclidean space to make them more similar, and hence improve the learning performance for a new subject. Our approach has three desirable properties: 1) it aligns the EEG trials directly in the Euclidean space, and any signal processing, …
Reason to Play: Behavioral and Brain Alignment Between Frontier LRMs and Human Game Learners
University of Oxford（牛津大学）；Columbia University（哥伦比亚大学）；New York University（纽约大学）；Massachusetts Institute of Technology（麻省理工学院）；Harvard University（哈佛大学）
Training-Driven Representational Geometry Modularization Predicts Brain Alignment in Language Models
School of Biomedical Engineering, Tsinghua University, Beijing, China（生物医学工程学院，清华大学，北京，中国）；Tsinghua Laboratory of Brain and Intelligence, Tsinghua University, Beijing, China（脑与智能实验室，清华大学，北京，中国）；School of Basic Medical Sciences, Tsinghua University, Beijing, China（基础医学学院，清华大学，北京，中国）；Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China（心理学与认知科学系，清华大学，北京，中国）
Retrieval-Based Brain Decoding by Alignment, not Complexity
University of Rome Tor Vergata（罗马托尔维加塔大学）；Tether Evo；A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital（麻省总医院 A.A. 马丁诺斯生物医学成像中心）
Université de Sousse（苏塞大学）；Ecole Nationale d’Ingénieurs de Sousse（苏塞国立工程师学院）；Istanbul Technical University（伊斯坦布尔理工大学）
Brain graph synthesis marked a new era for predicting a target brain graph from a source one without incurring the high acquisition cost and processing time of neuroimaging data. However, existing multi-modal graph synthesis frameworks have several limitations. First, they mainly focus on generating graphs from the same domain (intra-modality), overlooking the rich multimodal representations of brain connectivity (inter-modality). Second, they can only handle isomorphic graph generation tasks, limiting their generalizability to synthesizing target graphs with a different node size and topologi…
Istanbul Technical University（伊斯坦布尔理工大学）；Université de Sousse（苏塞大学）；Ecole Nationale d’Ingénieurs de Sousse（苏塞国立工程师学院）
Synthesizing multimodality medical data provides complementary knowledge and helps doctors make precise clinical decisions. Although promising, existing multimodal brain graph synthesis frameworks have several limitations. First, they mainly tackle only one problem (intra- or inter-modality), limiting their generalizability to synthesizing inter- and intra-modality simultaneously. Second, while few techniques work on super-resolving low-resolution brain graphs within a single modality (i.e., intra), inter-modality graph super-resolution remains unexplored though this would avoid the need for c…
Brain-like Flexible Visual Inference by Harnessing Feedback-Feedforward Alignment
Center for Theoretical Neuroscience（理论神经科学中心）；Zuckerman Mind Brain Behavior Institute（祖克曼心智大脑行为研究所）；Columbia University（哥伦比亚大学）；Department of Neuroscience（神经科学系）
Inter-subject registration of cortical areas is necessary in functional imaging (fMRI) studies for making inferences about equivalent brain function across a population. However, many high-level visual brain areas are defined as peaks of functional contrasts whose cortical position is highly variable. As such, most alignment methods fail to accurately map functional regions of interest (ROIs) across participants. To address this problem, we propose a locally optimized registration method that directly predicts the location of a seed ROI on a separate target cortical sheet by maximizing the fun…
Brain-like Functional Organization within Large Language Models
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Computing, University of Georgia（佐治亚大学计算学院）；School of Computer and Cyber Sciences, Augusta University（奥古斯塔大学计算机与网络科学学院）
BG-GAN: Generative AI Enable Representing Brain Structure-Function Connections for Alzheimer's Disease
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Stevens Institute of Technology（史蒂文斯理工学院）；Hong Kong Metropolitan University（香港都会大学）；VinUniversity（越南 VinUniversity 大学）；King Fahd University of Petroleum and Minerals（法赫德国王石油矿产大学）；Uppsala University（乌普萨拉大学）；Tiangong University（天津工业大学）
Cognitive flexibility describes the human ability to switch between modes of mental function to achieve goals. Mental switching is accompanied by transient changes in brain activity, which must occur atop an anatomical architecture that bridges disparate cortical and subcortical regions by underlying white matter tracts. However, an integrated perspective regarding how white matter networks might constrain brain dynamics during cognitive processes requiring flexibility has remained elusive. To address this challenge, we applied emerging tools from graph signal processing to decompose BOLD sign…
