BrainSCUBA：视觉皮层选择性的细粒度自然语言描述
BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity
AI总结：
BrainSCUBA 利用对比视觉-语言模型和大型语言模型，为视觉皮层体素生成细粒度自然语言描述，实现数据驱动的语义选择性刻画，并在身体选择性区域发现新的细粒度语义偏好。
摘要：
理解高级视觉皮层的功能组织是神经科学的一个核心焦点。过去的研究主要使用手工选择的刺激来绘制神经群体的视觉和语义选择性，这可能潜在地使结果偏向于关于视觉皮层功能的预先假设。超越传统方法，我们引入一种数据驱动的方法，为预测能最大程度激活感兴趣体素的图像生成自然语言描述。我们的方法——基于大脑对齐的语义描述（“BrainSCUBA”）——建立在对比视觉-语言模型学习到的丰富嵌入空间之上，并利用预训练的大型语言模型生成可解释的描述。我们通过在高级视觉区域进行细粒度的体素级描述来验证我们的方法。我们进一步使用这些描述进行文本条件图像合成，并表明我们的图像在语义上连贯且产生高预测激活。最后，为了展示我们的方法如何促进科学发现，我们对大脑中“人”表征的分布进行了探索性研究，并在身体选择性区域发现了细粒度的语义选择性。与早期解码文本的研究不同，我们的方法推导出体素级的语义选择性描述。我们的结果表明，BrainSCUBA 是理解大脑功能偏好的一种有前景的手段，并为进一步的假设驱动的视觉皮层研究提供了动力。
英文摘要：
Understanding the functional organization of higher visual cortex is a central focus in neuroscience. Past studies have primarily mapped the visual and semantic selectivity of neural populations using hand-selected stimuli, which may potentially bias results towards pre-existing hypotheses of visual cortex functionality. Moving beyond conventional approaches, we introduce a data-driven method that generates natural language descriptions for images predicted to maximally activate individual voxels of interest. Our method -- Semantic Captioning Using Brain Alignments ("BrainSCUBA") -- builds upon the rich embedding space learned by a contrastive vision-language model and utilizes a pre-trained large language model to generate interpretable captions. We validate our method through fine-grained voxel-level captioning across higher-order visual regions. We further perform text-conditioned image synthesis with the captions, and show that our images are semantically coherent and yield high predicted activations. Finally, to demonstrate how our method enables scientific discovery, we perform exploratory investigations on the distribution of "person" representations in the brain, and discover fine-grained semantic selectivity in body-selective areas. Unlike earlier studies that decode text, our method derives voxel-wise captions of semantic selectivity. Our results show that BrainSCUBA is a promising means for understanding functional preferences in the brain, and provides motivation for further hypothesis-driven investigation of visual cortex.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
