提出结合人工神经元分析与 fMRI 体素编码的框架，揭示 CLIP 和 METER 在神经元层面的类脑层级表征、冗余、极性及架构差异。
摘要：
尽管受脑启发的人工智能（AI）已展现出良好结果，但目前对人工神经网络（ANN）与人脑处理之间相似性的理解仍然有限：（1）单模态 ANN 研究无法捕捉大脑固有的多模态处理能力；（2）多模态 ANN 研究主要关注高层模型输出，而忽视了单个神经元的关键作用。为解决这些局限，我们提出一种新颖的神经元级分析框架，通过人类大脑活动来研究视觉-语言模型（VLM）中的多模态信息处理机制。我们的方法独特地将细粒度人工神经元（AN）分析与基于 fMRI 的体素编码相结合，考察两种架构不同的 VLM：CLIP 和 METER。分析揭示了四项关键发现：（1）AN 能成功预测多个功能网络（包括语言、视觉、注意和默认模式）中的生物神经元（BN）活动，表明二者具有共享的表征机制；（2）AN 和 BN 都通过重叠的神经表征表现出功能冗余，反映了大脑容错且协作的信息处理机制；（3）AN 呈现出与 BN 平行的极性模式，反向激活的 BN 在 VLM 各层中表现出镜像激活趋势，反映了神经信息处理的复杂性和双向性；（4）CLIP 与 METER 的架构驱动不同的 BN：CLIP 的独立分支表现出模态特化，而 METER 的跨模态设计产生统一的跨模态激活，突显架构对 ANN 类脑特性的影响。这些结果为 VLM 在神经元层面的类脑层级处理提供了有力证据。
英文摘要：
While brain-inspired artificial intelligence(AI) has demonstrated promising results, current understanding of the parallels between artificial neural networks (ANNs) and human brain processing remains limited: (1) unimodal ANN studies fail to capture the brain's inherent multimodal processing capabilities, and (2) multimodal ANN research primarily focuses on high-level model outputs, neglecting the crucial role of individual neurons. To address these limitations, we propose a novel neuron-level analysis framework that investigates the multimodal information processing mechanisms in vision-language models (VLMs) through the lens of human brain activity. Our approach uniquely combines fine-grained artificial neuron (AN) analysis with fMRI-based voxel encoding to examine two architecturally distinct VLMs: CLIP and METER. Our analysis reveals four key findings: (1) ANs successfully predict biological neurons (BNs) activities across multiple functional networks (including language, vision, attention, and default mode), demonstrating shared representational mechanisms; (2) Both ANs and BNs demonstrate functional redundancy through overlapping neural representations, mirroring the brain's fault-tolerant and collaborative information processing mechanisms; (3) ANs exhibit polarity patterns that parallel the BNs, with oppositely activated BNs showing mirrored activation trends across VLM layers, reflecting the complexity and bidirectional nature of neural information processing; (4) The architectures of CLIP and METER drive distinct BNs: CLIP's independent branches show modality-specific specialization, whereas METER's cross-modal design yields unified cross-modal activation, highlighting the architecture's influence on ANN brain-like properties. These results provide compelling evidence for brain-like hierarchical processing in VLMs at the neuronal level.
Stacked Regression using Off-the-shelf, Stimulus-tuned and Fine-tuned Neural Networks for Predicting fMRI Brain Responses to Movies (Algonauts 2025 Report)
Université Paris Cité（巴黎Cité大学）；University of Oxford（牛津大学）；University of Turin（都灵大学）；Universität Leipzig（莱比锡大学）；Max Planck School of Cognition（马克斯·普朗克认知科学学院）
fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding
School of Data Science, Fudan University（复旦大学数据科学学院）；Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学类脑智能科学与技术研究院）
MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Nanjing University of Science and Technology（南京理工大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
Seoul National University（首尔大学）；SungKyunKwan University（成均馆大学）；Brookhaven National Lab（布鲁克海文国家实验室）
Modeling spatiotemporal brain dynamics from high-dimensional data, such as functional Magnetic Resonance Imaging (fMRI), is a formidable task in neuroscience. Existing approaches for fMRI analysis utilize hand-crafted features, but the process of feature extraction risks losing essential information in fMRI scans. To address this challenge, we present SwiFT (Swin 4D fMRI Transformer), a Swin Transformer architecture that can learn brain dynamics directly from fMRI volumes in a memory and computation-efficient manner. SwiFT achieves this by implementing a 4D window multi-head self-attention mec…
A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli
Beijing Institute of Basic Medical Sciences（北京基础医学研究所）；School of Computer Science and Information Engineering, Hefei University of Technology (HFUT)（合肥工业大学计算机与信息工程学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Zhejiang University（浙江大学）；Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney（悉尼科技大学工程与信息技术学院澳大利亚人工智能研究所）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）
BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics
Department of Biomedical Engineering, Southern University of Science and Technology（南方科技大学生物医学工程系）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）
INRIA Saclay-Ile-de-France（法国国家信息与自动化研究所 萨克莱-法兰西岛分院）；INSERM（法国国家健康与医学研究院）；CEA（法国原子能与替代能源委员会）；Université Paris Sud（巴黎南大学）
We propose a method that combines signals from many brain regions observed in functional Magnetic Resonance Imaging (fMRI) to predict the subject's behavior during a scanning session. Such predictions suffer from the huge number of brain regions sampled on the voxel grid of standard fMRI data sets: the curse of dimensionality. Dimensionality reduction is thus needed, but it is often performed using a univariate feature selection procedure, that handles neither the spatial structure of the images, nor the multivariate nature of the signal. By introducing a hierarchical clustering of the brain v…
Chinese PLA General Hospital（中国人民解放军总医院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Peng Cheng Laboratory（鹏城实验室）；Beijing Jiaotong University（北京交通大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
In neuroimaging analysis, fMRI can well assess the function changes for brain diseases with no obvious structural lesions. To date, most deep-learning-based fMRI studies have employed functional connectivity (FC) as the basic feature for disease classification. However, FC is calculated on time series of predefined regions of interest and neglects detailed information contained in each voxel. Another drawback of using FC is the limited sample size for the training of deep models. The low representation ability of FC leads to poor performance in clinical practice, especially when dealing with m…
Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models
Department of Psychiatry and Behavioral Sciences, Stanford University（精神病学与行为科学系，斯坦福大学）；Department of Computer Science, Stanford University（计算机科学系，斯坦福大学）；Department of Neurobiology, University of Chicago（神经生物学系，芝加哥大学）；Google DeepMind（谷歌DeepMind）
Columbia University（哥伦比亚大学）；New York State Psychiatric Institute（纽约州精神病学研究所）
The goal of emotional brain state classification on functional MRI (fMRI) data is to recognize brain activity patterns related to specific emotion tasks performed by subjects during an experiment. Distinguishing emotional brain states from other brain states using fMRI data has proven to be challenging due to two factors: a difficulty to generate fast yet accurate predictions in short time frames, and a difficulty to extract emotion features which generalize to unseen subjects. To address these challenges, we conducted an experiment in which 22 subjects viewed pictures designed to stimulate ei…
BrainCast: A Spatio-Temporal Forecasting Model for Whole-Brain fMRI Time Series Prediction
Institute of Advanced Technology, University of Science and Technology of China（科学技术大学先进技术研究院）；MoE Key Laboratory of Brain-Inspired Intelligence Perception and Cognition, University of Science and Technology of China（科学技术大学脑启发智能感知与认知教育部重点实验室）；School of Mathematics and Computer Sciences, Nanchang University（南昌大学数学与计算机科学学院）；School of Data Science, University of Virginia（弗吉尼亚大学数据科学学院）；Department of Biomedical Engineering, Tulane University（路易斯安那大学医学工程系）
Middle East Technical University（中东技术大学）；Carnegie Mellon University（卡内基梅隆大学）
In this study, we propose a neural network approach to capture the functional connectivities among anatomic brain regions. The suggested approach estimates a set of brain networks, each of which represents the connectivity patterns of a cognitive process. We employ two different architectures of neural networks to extract directed and undirected brain networks from functional Magnetic Resonance Imaging (fMRI) data. Then, we use the edge weights of the estimated brain networks to train a classifier, namely, Support Vector Machines(SVM) to label the underlying cognitive process. We compare our b…
Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding
Gamania Digital Entertainment Co., Ltd.（Gamania数字娱乐有限公司）；Institute of Information Science, Academia Sinica（Academia Sinica信息科学研究所）；Sony Computer Science Laboratories, Inc.（索尼计算机科学实验室有限公司）
Generative AI Enables Structural Brain Network Construction from fMRI via Symmetric Diffusion Learning
Department of Computing, Hong Kong Polytechnic University（香港理工大学计算学系）；School of Information Engineering, Nanchang University（南昌大学信息工程学院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining
Department of Biomedical Engineering, Southern University of Science and Technology, China（南方科技大学生物医学工程系，中国）；School of Biomedical Engineering, Shenzhen University, China（深圳大学生物医学工程学院，中国）
National Kapodistrian Univ. of Athens (NKUA/UoA)（雅典国家卡波季斯特里安大学）
Functional Magnetic Resonance Imaging (fMRI) is a powerful non-invasive tool for localizing and analyzing brain activity. This study focuses on one very important aspect of the functional properties of human brain, specifically the estimation of the level of parallelism when performing complex cognitive tasks. Using fMRI as the main modality, the human brain activity is investigated through a purely data-driven signal processing and dimensionality analysis approach. Specifically, the fMRI signal is treated as a multi-dimensional data space and its intrinsic `complexity' is studied via dataset …
University of Rome, Tor Vergata（罗马第二大学）；Istituto di Bioimmagini e Fisiologia Molecolare, CNR（意大利国家研究委员会生物成像与分子生理学研究所）；Martinos Center for Biomedical Imaging（马丁诺斯生物医学成像中心）；MGH（麻省总医院）；Harvard Medical School（哈佛医学院）
Brain decoding is a field of computational neuroscience that uses measurable brain activity to infer mental states or internal representations of perceptual inputs. Therefore, we propose a novel approach to brain decoding that also relies on semantic and contextual similarity. We employ an fMRI dataset of natural image vision and create a deep learning decoding pipeline inspired by the existence of both bottom-up and top-down processes in human vision. We train a linear brain-to-feature model to map fMRI activity features to visual stimuli features, assuming that the brain projects visual info…
In the past few years, significant advancements were made in reconstruction of observed natural images from fMRI brain recordings using deep-learning tools. Here, for the first time, we show that dense 3D depth maps of observed 2D natural images can also be recovered directly from fMRI brain recordings. We use an off-the-shelf method to estimate the unknown depth maps of natural images. This is applied to both: (i) the small number of images presented to subjects in an fMRI scanner (images for which we have fMRI recordings - referred to as "paired" data), and (ii) a very large number of natura…
Functional Magnetic Resonance Images acquired during resting-state provide information about the functional organization of the brain through measuring correlations between brain areas. Independent components analysis is the reference approach to estimate spatial components from weakly structured data such as brain signal time courses; each of these components may be referred to as a brain network and the whole set of components can be conceptualized as a brain functional atlas. Recently, new methods using a sparsity prior have emerged to deal with low signal-to-noise ratio data. However, even…
Yale University（耶鲁大学）；Yale School of Medicine（耶鲁医学院）
Autism spectrum disorder (ASD) is a complex neurodevelopmental disorder. Finding the biomarkers associated with ASD is extremely helpful to understand the underlying roots of the disorder and can lead to earlier diagnosis and more targeted treatment. Although Deep Neural Networks (DNNs) have been applied in functional magnetic resonance imaging (fMRI) to identify ASD, understanding the data-driven computational decision making procedure has not been previously explored. Therefore, in this work, we address the problem of interpreting reliable biomarkers associated with identifying ASD; specific…
4D iterative reconstruction of brain fMRI in the moving fetus
Computational Imaging Research Lab（计算成像研究实验室）；Medical University of Vienna（维也纳医科大学）；Lausanne University Hospital (CHUV)（洛桑大学医院（CHUV））；University of Lausanne (UNIL)（洛桑大学（UNIL））；CIBM Center for Biomedical Imaging（CIBM生物医学成像中心）
Hi-DREAM: Brain-Inspired Hierarchical Diffusion for fMRI-to-Image Reconstruction via ROI Encoder and VisuAl Mapping
Dept of Data Science and AI, Monash University, Melbourne, Australia（数据科学与人工智能系，墨尔本大学，澳大利亚）；School of Psychological Sciences, Monash University, Melbourne, Australia（心理学科学学院，墨尔本大学，澳大利亚）；Brain and Mind Centre, University of Sydney, Sydney, Australia（脑与心智中心，悉尼大学，澳大利亚）
Tokyo Institute of Technology（东京工业大学）；National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）；AIST-Tokyo Tech Real World Big-Data Computation Open Innovation Laboratory（AIST-东京工业大学真实世界大数据计算开放创新实验室）
With the advent of brain imaging techniques and machine learning tools, much effort has been devoted to building computational models to capture the encoding of visual information in the human brain. One of the most challenging brain decoding tasks is the accurate reconstruction of the perceived natural images from brain activities measured by functional magnetic resonance imaging (fMRI). In this work, we survey the most recent deep learning methods for natural image reconstruction from fMRI. We examine these methods in terms of architectural design, benchmark datasets, and evaluation metrics …
Reconstructing natural videos from fMRI brain recordings is very challenging, for two main reasons: (i) As fMRI data acquisition is difficult, we only have a limited amount of supervised samples, which is not enough to cover the huge space of natural videos; and (ii) The temporal resolution of fMRI recordings is much lower than the frame rate of natural videos. In this paper, we propose a self-supervised approach for natural-movie reconstruction. By employing cycle-consistency over Encoding-Decoding natural videos, we can: (i) exploit the full framerate of the training videos, and not be limit…
SLIM-Brain: A Data- and Training-Efficient Foundation Model for fMRI Data Analysis
Department of Biomedical Engineering, Southern University of Science and Technology, China（南方科技大学生物医学工程系）；Department of Computer Science, University of Warwick, The UK（沃里克大学计算机科学系）；Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, China（复旦大学脑启发式智能科学技术研究所）
While medical imaging typically provides massive amounts of data, the extraction of relevant information for predictive diagnosis remains a difficult challenge. Functional MRI (fMRI) data, that provide an indirect measure of task-related or spontaneous neuronal activity, are classically analyzed in a mass-univariate procedure yielding statistical parametric maps. This analysis framework disregards some important principles of brain organization: population coding, distributed and overlapping representations. Multivariate pattern analysis, i.e., the prediction of behavioural variables from brai…
Reconstructing perceived natural images from fMRI signals is one of the most engaging topics of neural decoding research. Prior studies had success in reconstructing either the low-level image features or the semantic/high-level aspects, but rarely both. In this study, we utilized an Instance-Conditioned GAN (IC-GAN) model to reconstruct images from fMRI patterns with both accurate semantic attributes and preserved low-level details. The IC-GAN model takes as input a 119-dim noise vector and a 2048-dim instance feature vector extracted from a target image via a self-supervised learning model (…
