arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.11737cs.CVcs.LGq-bio.NC
收藏

人脑中的视觉表征与大语言模型对齐

Visual representations in the human brain are aligned with large language models

Institute of Cognitive Science, University of Osnabrück(奥斯纳布吕克大学认知科学研究所);Donders Institute for Brain, Cognition & Behaviour(唐德斯脑、认知与行为研究所);Department of Psychology, University of Minnesota(明尼苏达大学心理学系);Graduate Program in Cognitive Science, University of Minnesota(明尼苏达大学认知科学研究生项目);Department of Neuroscience, University of Minnesota(明尼苏达大学神经科学系);Université de Montréal(蒙特利尔大学);Center for Human Brain Health, School of Psychology, University of Birmingham(伯明翰大学心理学院人类大脑健康中心)

Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, Ian Charest

AI总结:

该研究证明大语言模型对场景描述的嵌入能有效建模人脑视觉活动,且可重建场景描述,并发现将图像转换为LLM表征的深度网络比众多最先进模型更符合大脑表征。

摘要:

人脑从视觉输入中提取复杂信息,包括物体、它们的空间和语义相互关系,以及它们与环境的交互。然而,用于研究这些信息的定量方法仍然缺乏。在这里,我们测试了大语言模型(LLMs)中编码的上下文信息是否有助于建模人脑从自然场景中提取的复杂视觉信息。我们表明,场景描述的LLM嵌入能够成功地表征观看自然场景所诱发的大脑活动。这种映射捕捉了不同脑区的选择性,并且足够稳健,以至于可以从大脑活动中重建准确的场景描述。通过精心控制的模型比较,我们进一步表明,LLM表征与大脑表征匹配的准确性源于LLM整合场景描述中所包含的复杂信息的能力,这些信息超出了单个词语所传达的内容。最后,我们训练深度神经网络模型将图像输入转换为LLM表征。值得注意的是,尽管这些网络使用的训练数据比大量最先进的替代模型少几个数量级,但它们学习到的表征比这些模型更符合大脑表征。总体而言,我们的结果表明,场景描述的LLM嵌入提供了一种表征格式,能够解释大脑从视觉输入中提取的复杂信息。

英文摘要:

The human brain extracts complex information from visual inputs, including objects, their spatial and semantic interrelations, and their interactions with the environment. However, a quantitative approach for studying this information remains elusive. Here, we test whether the contextual information encoded in large language models (LLMs) is beneficial for modelling the complex visual information extracted by the brain from natural scenes. We show that LLM embeddings of scene captions successfully characterise brain activity evoked by viewing the natural scenes. This mapping captures selectivities of different brain areas, and is sufficiently robust that accurate scene captions can be reconstructed from brain activity. Using carefully controlled model comparisons, we then proceed to show that the accuracy with which LLM representations match brain representations derives from the ability of LLMs to integrate complex information contained in scene captions beyond that conveyed by individual words. Finally, we train deep neural network models to transform image inputs into LLM representations. Remarkably, these networks learn representations that are better aligned with brain representations than a large number of state-of-the-art alternative models, despite being trained on orders-of-magnitude less data. Overall, our results suggest that LLM embeddings of scene captions provide a representational format that accounts for complex information extracted by the brain from visual inputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑