Research has repeatedly demonstrated that intermediate hidden states extracted from large language models and speech audio models predict measured brain response to natural language stimuli. Yet, very little is known about the representation properties that enable this high prediction performance. Why is it the intermediate layers, and not the output layers, that are most effective for this unique and highly general transfer task? We give evidence that the correspondence between speech and language models and the brain derives from shared meaning abstraction and not their next-word prediction properties. In particular, models construct higher-order linguistic features in their middle layers, cued by a peak in the layerwise intrinsic dimension, a measure of feature complexity. We show that a layer's intrinsic dimension strongly predicts how well it explains fMRI and ECoG signals; that the relation between intrinsic dimension and brain predictivity arises over model pre-training; and finetuning models to better predict the brain causally increases both representations' intrinsic dimension and their semantic content. Results suggest that semantic richness, high intrinsic dimension, and brain predictivity mirror each other, and that the key driver of model-brain similarity is rich meaning abstraction of the inputs, where language modeling is a task complex enough (but perhaps not the only) to require it.
RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain
Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）
Conditioning Large Language Models on Legal Systems? Detecting Punishable Hate Speech
Central Office for Information Technology in the Security Sector (ZITiS)（安全领域信息技术中央办公室（ZITiS））；Computational Linguistics, CATALPA, FernUniversität in Hagen（计算语言学、CATALPA、费尔本大学）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；Waseda Institute for Advanced Study, Waseda University（早稻田大学高级研究机构）
Improve Language Model and Brain Alignment via Associative Memory
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（人工智能学院，南京航空航天大学）；The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education（脑机智能技术重点实验室，教育部）
Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech
Facebook AI Research（Facebook AI研究院）；Université de Paris（巴黎大学）；CoML, ENS/CNRS/EHESS/INRIA/PSL University（计算语言学实验室，巴黎高等师范学院/法国国家科学研究中心/法国社会科学高等研究院/法国国家信息与自动化研究所/巴黎文理研究大学）；CRI, Département Frontières du Vivant et de l’Apprendre, IIFR, Université de Paris（认知研究所，生命与学习前沿系，法国医学研究院，巴黎大学）；École normale supérieure, PSL University, CNRS（巴黎高等师范学院，巴黎文理研究大学，法国国家科学研究中心）
Toward a realistic model of speech processing in the brain with self-supervised learning
Ecole Normale Supérieure, PSL University（巴黎高等师范学院，巴黎文理研究大学）；Université de Paris cité（巴黎城市大学）；Université Paris-Saclay（巴黎萨克雷大学）；Inria（法国国家信息与自动化研究所）；CEA（法国原子能和替代能源委员会）；University of Toronto（多伦多大学）；INSERM（法国国家健康与医学研究院）；Meta AI
An Approach to Simultaneous Acquisition of Real-Time MRI Video, EEG, and Surface EMG for Articulatory, Brain, and Muscle Activity During Speech Production
Signal Analysis and Interpretation Laboratory, University of Southern California（南加州大学信号分析与解释实验室）；Ming Hsieh Dept. of Electrical and Computer Engineering, University of Southern California（南加州大学明希斯电气与计算机工程系）；Dept. of Linguistics, University of Southern California（南加州大学语言学系）
Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals
Peking University（北京大学）；Chinese Institute for Brain Research（中国脑科学研究院）；Beijing Normal University（北京师范大学）；Shanghai Jiao Tong University（上海交通大学）；Capital Medical University（首都医科大学）；Xuanwu Hospital（宣武医院）
MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data
Graduate School of Humanities and Sciences Ochanomizu University（大仓山大学人文科学研究生院）；Center for Information and Neural Networks Advanced ICT Research Institute National Institute of Information and Communications Technology（信息与神经网络中心先进信息通信研究机构信息通信技术研究所）
Brain-to-Language Decoding: Paradigms, Signals, Methods, Evaluation, Practical Use and Beyond
Uploading Inc；Human-centric Artificial Intelligence Centre, University of Technology Sydney（悉尼科技大学以人为本人工智能中心）；Stanford Institute for Human-Centered Artificial Intelligence (HAI), Stanford University（斯坦福大学以人为本人工智能研究所）
Decoding Phone Pairs from MEG Signals Across Speech Modalities
HiTZ Center, University of the Basque Country -- UPV/EHU（巴斯克国家大学希茨中心）；Basque Centre on Cognition, Brain and Language -- BCBL（巴斯克认知、大脑与语言中心）；Laboratory of Functional Anatomy, Faculty of Human Motor Sciences, Université libre de Bruxelles (ULB)（人类运动科学系功能解剖实验室）；Laboratoire de Neuroanatomie et Neuroimagerie translationnelles (LN2T), ULB Neuroscience Institute, Université libre de Bruxelles (ULB)（神经解剖与神经影像翻译实验室（LN2T）、ULB神经科学研究所）；WEL Research Institute（WEL研究机构）
Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
Inria Bordeaux（法国国家信息与自动化研究所波尔多分所）；LaBRI, Université de Bordeaux（波尔多大学拉布里实验室）；Université de Bordeaux（波尔多大学）；TU Berlin（柏林工业大学）；National University of Singapore（新加坡国立大学）；Microsoft（微软公司）；International Institute of Information Technology Hyderabad（海得拉巴国际信息技术研究所）；CEA（法国原子能和替代能源委员会）；CNRS（法国国家科学研究中心）
