Visual representations in the human brain are aligned with large language models
Institute of Cognitive Science, University of Osnabrück(奥斯纳布吕克大学认知科学研究所);Donders Institute for Brain, Cognition & Behaviour(唐德斯脑、认知与行为研究所);Department of Psychology, University of Minnesota(明尼苏达大学心理学系);Graduate Program in Cognitive Science, University of Minnesota(明尼苏达大学认知科学研究生项目);Department of Neuroscience, University of Minnesota(明尼苏达大学神经科学系);Université de Montréal(蒙特利尔大学);Center for Human Brain Health, School of Psychology, University of Birmingham(伯明翰大学心理学院人类大脑健康中心)
Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, Ian Charest
The human brain extracts complex information from visual inputs, including objects, their spatial and semantic interrelations, and their interactions with the environment. However, a quantitative approach for studying this information remains elusive. Here, we test whether the contextual information encoded in large language models (LLMs) is beneficial for modelling the complex visual information extracted by the brain from natural scenes. We show that LLM embeddings of scene captions successfully characterise brain activity evoked by viewing the natural scenes. This mapping captures selectivities of different brain areas, and is sufficiently robust that accurate scene captions can be reconstructed from brain activity. Using carefully controlled model comparisons, we then proceed to show that the accuracy with which LLM representations match brain representations derives from the ability of LLMs to integrate complex information contained in scene captions beyond that conveyed by individual words. Finally, we train deep neural network models to transform image inputs into LLM representations. Remarkably, these networks learn representations that are better aligned with brain representations than a large number of state-of-the-art alternative models, despite being trained on orders-of-magnitude less data. Overall, our results suggest that LLM embeddings of scene captions provide a representational format that accounts for complex information extracted by the brain from visual inputs.
CLIMB: Controllable Longitudinal Brain Image Generation using Mamba-based Latent Diffusion Model and Gaussian-aligned Autoencoder
Department of Artificial Intelligence Convergence, Chonnam National University(全南国立大学人工智能融合系);Department of Nuclear Medicine, Chonnam National University(全南国立大学核医学系);University of South Australia(南澳大利亚大学);AdelaideMRI(阿德莱德MRI);Electronics and Telecommunications Research Institute(电子电信研究所)
SEED: Towards More Accurate Semantic Evaluation for Visual Brain Decoding
IPAI, Seoul National University(韩亚国立大学IPAI);ECE, Seoul National University(韩亚国立大学电子工程系);Psychology, Seoul National University(韩亚国立大学心理学系);Brookhaven National Lab(布鲁赫斯国家实验室);ASRI / INMC / AIIS, Seoul National University(韩亚国立大学ASRI/INMC/AIIS)
Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli
University of Rome, Tor Vergata(罗马第二大学);Istituto di Bioimmagini e Fisiologia Molecolare, CNR(意大利国家研究委员会生物成像与分子生理学研究所);Martinos Center for Biomedical Imaging(马丁诺斯生物医学成像中心);MGH(麻省总医院);Harvard Medical School(哈佛医学院)
ViBE: Visual-to-M/EEG Brain Encoding via Spatio-Temporal VAE and Distribution-Aligned Projection
Jinan University(济南大学);The First Affiliated Hospital of Jinan University(济南大学第一附属医院);Western University(西方大学);Guangdong University of Technology(广东工业大学);Tsinghua University(清华大学)
Achieving More Human Brain-Like Vision via Human EEG Representational Alignment
Department of Psychology, The Ohio State University(心理学系,俄亥俄州立大学);McGovern Institute for Brain Research, Massachusetts Institute of Technology(脑科学研究所,麻省理工学院);Department of Neuroscience, The University of Texas at Dallas(神经科学系,德克萨斯大学达拉斯分校)
BRACTIVE: A Brain Activation Approach to Human Visual Brain Learning
Electrical Engineering & Computer Science Department, University of Arkansas(亚拉巴马大学电气工程与计算机科学系);Brain and Cognitive Engineering, Korea University(韩国大学脑科学与认知工程系);Department of Computer Science, University at Albany(阿尔巴尼大学计算机科学系);Mike Wiegers Department of Electrical & Computer Engineering, Kansas State University(堪萨斯州立大学迈克·魏格斯电气与计算机工程系);Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology(麻省理工学院脑科学与认知科学系)
MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Nanjing University of Science and Technology(南京理工大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学)
Brain2Text Decoding Model Reveals the Neural Mechanisms of Visual Semantic Processing
Ministry of Education Center for Studies of Psychological Application(教育部心理应用研究中心);Center for Studies of Psychological Application(心理应用研究中心);Key Laboratory of Brain, Cognition and Education Sciences(脑认知与教育科学重点实验室);School of Psychology(心理学学院);Guangdong Key Laboratory of Mental Health and Cognitive Science(广东省心理健康与认知科学重点实验室)
Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models
Department of Psychiatry and Behavioral Sciences, Stanford University(精神病学与行为科学系,斯坦福大学);Department of Computer Science, Stanford University(计算机科学系,斯坦福大学);Department of Neurobiology, University of Chicago(神经生物学系,芝加哥大学);Google DeepMind(谷歌DeepMind)
MindAdapter: Few-Shot Parameter-Efficient Residual Calibration of Cross-Subject Brain-to-Visual Decoding Models
Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);Agency for Science, Technology and Research(科技研究局);New York University(纽约大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Department of Computer and Information Science, University of Macau(澳门大学计算机与信息科学系)
Cogni-Net: Cognitive Feature Learning through Deep Visual Perception
Nanyang Technological University(南洋理工大学);Indian Institute of Technology Roorkee(印度理工学院鲁尔基分校)
我们能让计算机识别我们仅仅从大脑信号中看到的东西吗?我们的论文旨在利用流行的预训练的视觉模型在视觉领域学习的知识,并用它来教一个经常性的模型被训练的大脑信号学习的区别性的多方面的人类大脑的认知不同的视觉对象类别响应感知的视觉线索。为此,我们利用从视觉刺激(如图像)触发的脑EEG信号,并利用图像及其相应的脑信号之间的自然同步来学习认知特征空间的新表示。知识蒸馏的概念已在这里用于训练的深层认知模型,认知网\footnote{The source code of the proposed system is publicly available at {https://www.github.com/53X/CogniNET}},采用学生-教师学习技术,以桥梁的过程中,跨模态的知识转移。所提出的新架构获得了最先进的结果,显着超过其他现有的模型。我们进行的实验还表明,如果可以大规模收集大脑EEG信号等视觉刺激信息,那么这将有助于更好地了解人类大脑认知的大部分未开发领域。
Tongji University(同济大学);University of Oxford(牛津大学);North China Institute of Computing Technology, State Information Center of China(国家信息中心华北计算技术研究所);Macquarie University(麦考瑞大学)