语言调节视觉：来自神经网络和人类脑损伤模型的证据
Language modulates vision: Evidence from neural networks and human brain-lesion models
AI总结：
研究通过神经网络与脑损伤数据对比，揭示语言对视觉感知的调节作用，发现CLIP模型在VOTC活动解释上优于其他模型，且语言区域损伤影响模型与脑活动的一致性。
摘要：
比较深度神经网络（DNNs）和人类大脑的信息结构已成为探索其相似性和差异性的关键方法。近期研究表明，视觉-语言DNN模型（如CLIP）在人类腹侧枕颞叶皮层（VOTC）活动方面的对齐优于早期视觉模型，支持语言调节人类视觉感知的观点。然而，由于DNNs的“黑箱”性质，解释此类比较结果存在固有局限。为此，我们结合模型-大脑适应性分析与人类脑损伤数据，研究破坏视觉与语言系统通信路径如何因果影响视觉-语言DNN解释VOTC活动的能力。在四个多样化数据集中，CLIP consistently捕获了VOTC神经表示的独特方差，相对于标签监督（ResNet）和无监督（MoCo）模型。这一优势在群体层面倾向于左半球，与人类语言网络一致。对33名中风患者分析显示，VOTC与左角回语言区域之间白质完整性减少与CLIP-大脑一致性降低和MoCo-大脑一致性增加相关，表明语言处理动态影响VOTC活动。这些发现支持在神经认知模型中整合语言调节，强化视觉-语言DNN模型的概念。模型-大脑相似性对特定脑损伤的敏感性表明，利用人类脑部操纵是评估和开发类脑计算机模型的有前途的框架。
英文摘要：
Comparing information structures in between deep neural networks (DNNs) and the human brain has become a key method for exploring their similarities and differences. Recent research has shown better alignment of vision-language DNN models, such as CLIP, with the activity of the human ventral occipitotemporal cortex (VOTC) than earlier vision models, supporting the idea that language modulates human visual perception. However, interpreting the results from such comparisons is inherently limited due to the "black box" nature of DNNs. To address this, we combined model-brain fitness analyses with human brain lesion data to examine how disrupting the communication pathway between the visual and language systems causally affects the ability of vision-language DNNs to explain the activity of the VOTC. Across four diverse datasets, CLIP consistently captured unique variance in VOTC neural representations, relative to both label-supervised (ResNet) and unsupervised (MoCo) models. This advantage tended to be left-lateralized at the group level, aligning with the human language network. Analyses of 33 stroke patients revealed that reduced white matter integrity between the VOTC and the language region in the left angular gyrus was correlated with decreased CLIP-brain correspondence and increased MoCo-brain correspondence, indicating a dynamic influence of language processing on the activity of the VOTC. These findings support the integration of language modulation in neurocognitive models of human vision, reinforcing concepts from vision-language DNN models. The sensitivity of model-brain similarity to specific brain lesions demonstrates that leveraging manipulation of the human brain is a promising framework for evaluating and developing brain-like computer models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
