利用图像-语言基础模型识别人脑中共享的可解码概念
Identifying Shared Decodable Concepts in the Human Brain Using Image-Language Foundation Models
AI总结：
本文提出数据驱动方法，将fMRI脑反应映射至CLIP空间，并通过对比降维与体素掩蔽优化定位跨受试者共享的可解码语义概念。
摘要：
我们提出了一种利用高质量预训练多模态表征来探索人脑中细粒度语义网络的方法。以往研究已记录了大脑功能定位的证据，即不同解剖区域会优先对不同类型的感觉输入产生激活。许多此类局部结构已为人所知，包括梭状回面孔区和海马旁回位置区。这引出了一个问题：是否还有其他脑区（或脑区的联合）也专门用于处理其他重要的语义概念。为识别这些脑区，我们开发了一种数据驱动方法，从大规模功能性磁共振成像（fMRI）数据集中揭示可解码的视觉概念。我们的分析大致分为三个部分。首先，训练一个全连接神经网络，将大脑反应映射到图像-语言基础模型 CLIP（Radford 等，2021）的输出。随后，采用对比学习降维方法揭示 CLIP 空间中可被大脑解码的成分。在分析的最后一部分，我们使用体素掩蔽优化方法在大脑中定位共享的可解码概念，从而生成共享可解码概念（SDC）空间。通过与以往识别面孔、身体和位置区域的定位实验进行比较，我们验证了该流程的准确性。除了这些对应脑区已知的概念之外，我们还将跨受试者共享的新颖概念表征定位到人脑的其他区域。我们还展示了如何使用该方法检查单个受试者的细粒度语义网络。我们设想，这种可扩展方法也可调整用于探索人工智能与神经科学交叉领域的其他问题。
英文摘要：
We introduce a method that takes advantage of high-quality pretrained multimodal representations to explore fine-grained semantic networks in the human brain. Previous studies have documented evidence of functional localization in the brain, with different anatomical regions preferentially activating for different types of sensory input. Many such localized structures are known, including the fusiform face area and parahippocampal place area. This raises the question of whether additional brain regions (or conjunctions of brain regions) are also specialized for other important semantic concepts. To identify such brain regions, we developed a data-driven approach to uncover visual concepts that are decodable from a massive functional magnetic resonance imaging (fMRI) dataset. Our analysis is broadly split into three sections. First, a fully connected neural network is trained to map brain responses to the outputs of an image-language foundation model, CLIP (Radford et al., 2021). Subsequently, a contrastive-learning dimensionality reduction method reveals the brain-decodable components of CLIP space. In the final section of our analysis, we localize shared decodable concepts in the brain using a voxel-masking optimization method to produce a shared decodable concept (SDC) space. The accuracy of our procedure is validated by comparing it to previous localization experiments that identify regions for faces, bodies, and places. In addition to these concepts, whose corresponding brain regions were already known, we localize novel concept representations which are shared across participants to other areas of the human brain. We also demonstrate how this method can be used to inspect fine-grained semantic networks for individual participants. We envisage that this extensible method can also be adapted to explore other questions at the intersection of AI and neuroscience.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
