Representational Similarity Analysis (RSA) aims to explore similarities between neural activities of different stimuli. Classical RSA techniques employ the inverse of the covariance matrix to explore a linear model between the neural activities and task events. However, calculating the inverse of a large-scale covariance matrix is time-consuming and can reduce the stability and robustness of the final analysis. Notably, it becomes severe when the number of samples is too large. For facing this shortcoming, this paper proposes a novel RSA method called gradient-based RSA (GRSA). Moreover, the proposed method is not restricted to a linear model. In fact, there is a growing interest in finding more effective ways of using multi-subject and whole-brain fMRI data. Searchlight technique can extend RSA from the localized brain regions to the whole-brain regions with smaller memory footprint in each process. Based on Searchlight, we propose a new method called Spatiotemporal Searchlight GRSA (SSL-GRSA) that generalizes our ROI-based GRSA algorithm to the whole-brain data. Further, our approach can handle some computational challenges while dealing with large-scale, multi-subject fMRI data. Experimental studies on multi-subject datasets confirm that both proposed approaches achieve superior performance to other state-of-the-art RSA algorithms.
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Alberta（阿尔伯塔大学）；Purdue University Fort Wayne（普渡大学韦恩堡分校）
Similarity analysis is one of the crucial steps in most fMRI studies. Representational Similarity Analysis (RSA) can measure similarities of neural signatures generated by different cognitive states. This paper develops Deep Representational Similarity Learning (DRSL), a deep extension of RSA that is appropriate for analyzing similarities between various cognitive tasks in fMRI datasets with a large number of subjects, and high-dimensionality -- such as whole-brain images. Unlike the previous methods, DRSL is not limited by a linear transformation or a restricted fixed nonlinear kernel functio…
There is a growing interest in joint multi-subject fMRI analysis. The challenge of such analysis comes from inherent anatomical and functional variability across subjects. One approach to resolving this is a shared response factor model. This assumes a shared and time synchronized stimulus across subjects. Such a model can often identify shared information, but it may not be able to pinpoint with high resolution the spatial location of this information. In this work, we examine a searchlight based shared response model to identify shared information in small contiguous regions (searchlights) a…
Information mapping is a popular application of Multivoxel Pattern Analysis (MVPA) to fMRI. Information maps are constructed using the so called searchlight method, where the spherical multivoxel neighborhood of every voxel (i.e., a searchlight) in the brain is evaluated for the presence of task-relevant response patterns. Despite their widespread use, information maps present several challenges for interpretation. One such challenge has to do with inferring the size and shape of a multivoxel pattern from its signature on the information map. To address this issue, we formally examined the geo…
Multi-voxel pattern analysis (MVPA) is a fruitful and increasingly popular complement to traditional univariate methods of analyzing neuroimaging data. We propose to replace the standard 'decoding' approach to searchlight-based MVPA, measuring the performance of a classifier by its accuracy, with a method based on the multivariate form of the general linear model. Following the well-established methodology of multivariate analysis of variance (MANOVA), we define a measure that directly characterizes the structure of multi-voxel data, the pattern distinctness $D$. Our measure is related to stan…
Domain Adaptation-Enhanced Searchlight: Enabling classification of brain states from visual perception to mental imagery
University of the Basque Country (UPV/EHU)（巴斯克大学（UPV/EHU））；Basque Center for Cognition, Brain and Language (BCBL)（巴斯克认知、大脑与语言中心）；Ikerbasque, Basque Foundation for Science（伊克尔巴斯克基金会（巴斯克科学基金会））
Bruno Kessler Foundation（布鲁诺·凯斯勒基金会）；University of Trento（特伦托大学）；UC Berkeley（加州大学伯克利分校）
Determining the extent to which different cognitive modalities (understood here as the set of cognitive processes underlying the elaboration of a stimulus by the brain) rely on overlapping neural representations is a fundamental issue in cognitive neuroscience. In the last decade, the identification of shared activity patterns has been mostly framed as a supervised learning problem. For instance, a classifier is trained to discriminate categories (e.g. faces vs. houses) in modality I (e.g. perception) and tested on the same categories in modality II (e.g. imagery). This type of analysis is oft…
Directorate of Technology and Innovation（技术与创新局）；Chamber of Deputies（众议院）
Deep predictive coding networks are neuroscience-inspired unsupervised learning models that learn to predict future sensory states. We build upon the PredNet implementation by Lotter, Kreiman, and Cox (2016) to investigate if predictive coding representations are useful to predict brain activity in the visual cortex. We use representational similarity analysis (RSA) to compare PredNet representations to functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data from the Algonauts Project. In contrast to previous findings in the literature (Khaligh-Razavi &Kriegeskorte, …
The University of Tokyo（东京大学）；Princeton University（普林斯顿大学）；Facebook Reality Labs（Facebook现实实验室）；Columbia University（哥伦比亚大学）；Intel Corporation（英特尔公司）
With the wide adoption of functional magnetic resonance imaging (fMRI) by cognitive neuroscience researchers, large volumes of brain imaging data have been accumulated in recent years. Aggregating these data to derive scientific insights often faces the challenge that fMRI data are high-dimensional, heterogeneous across people, and noisy. These challenges demand the development of computational tools that are tailored both for the neuroscience questions and for the properties of the data. We review a few recently developed algorithms in various domains of fMRI research: fMRI in naturalistic ta…
Flexibility and Invariance of Object Representations in the Human Brain
Neuroscience Institute, Carnegie Mellon University（卡内基梅隆大学神经科学研究所）；Department of Psychology, Carnegie Mellon University（卡内基梅隆大学心理学系）；Department of Machine Learning, Carnegie Mellon University（卡内基梅隆大学机器学习系）；Department of Neurosurgery, University of Rochester Medical Center（罗切斯特大学医学中心神经外科）
Multivariate analysis of fMRI data has benefited substantially from advances in machine learning. Most recently, a range of probabilistic latent variable models applied to fMRI data have been successful in a variety of tasks, including identifying similarity patterns in neural data (Representational Similarity Analysis and its empirical Bayes variant, RSA and BRSA; Intersubject Functional Connectivity, ISFC), combining multi-subject datasets (Shared Response Mapping; SRM), and mapping between brain and behavior (Joint Modeling). Although these methods share some underpinnings, they have been d…
In this paper I review the submission to the Explaining the Human Visual Brain Challenge 2019 in both the fMRI and MEG tracks. The goal was to construct neural network features which generate the so-called representational dissimilarity matrix (RDM) which is most similar to the one extracted from fMRI and MEG data upon viewing a set of images. I review exploring the optimal granularity of the receptive field, a construction of intermediate surrogate features using Multidimensional Scaling and modelling them using neural network features. I also point out some peculiarities of the RDM construct…
Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining
Department of Biomedical Engineering, Southern University of Science and Technology, China（南方科技大学生物医学工程系，中国）；School of Biomedical Engineering, Shenzhen University, China（深圳大学生物医学工程学院，中国）
Untrained CNNs Exceed Backpropagation in V1 Alignment at High Evaluation Resolution: A Systematic RSA Comparison of Four Learning Rules Against Human fMRI
Net2Brain: A Toolbox to compare artificial vision models with human brain responses
Department of Computer Science, Goethe Universität（冯·迪斯大学计算机科学系）；Department of Education and Psychology, Freie Universität Berlin（柏林自由大学教育与心理学系）；Berlin School of Mind and Brain, Faculty of Philosophy（柏林心灵与大脑学校，哲学学院）；Bernstein Center for Computational Neuroscience Berlin（柏林计算神经科学中心）
Institute for Logic, Language and Computation（逻辑、语言与计算研究所）；University of Amsterdam（阿姆斯特丹大学）
In this paper, we define and apply representational stability analysis (ReStA), an intuitive way of analyzing neural language models. ReStA is a variant of the popular representational similarity analysis (RSA) in cognitive neuroscience. While RSA can be used to compare representations in models, model components, and human brains, ReStA compares instances of the same model, while systematically varying single model parameter. Using ReStA, we study four recent and successful neural language models, and evaluate how sensitive their internal representations are to the amount of prior context. Us…
Training-Driven Representational Geometry Modularization Predicts Brain Alignment in Language Models
School of Biomedical Engineering, Tsinghua University, Beijing, China（生物医学工程学院，清华大学，北京，中国）；Tsinghua Laboratory of Brain and Intelligence, Tsinghua University, Beijing, China（脑与智能实验室，清华大学，北京，中国）；School of Basic Medical Sciences, Tsinghua University, Beijing, China（基础医学学院，清华大学，北京，中国）；Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China（心理学与认知科学系，清华大学，北京，中国）
High-resolution functional imaging is providing increasingly rich measurements of brain activity in animals and humans. A major challenge is to leverage such data to gain insight into the brain's computational mechanisms. The first step is to define candidate brain-computational models (BCMs) that can perform the behavioural task in question. We would then like to infer, which of the candidate BCMs best accounts for measured brain-activity data. Here we describe a method that complements each BCM by a measurement model (MM), which simulates the way the brain-activity measurements reflect neuro…
What information from an act of sentence understanding is robustly represented in the human brain? We investigate this question by comparing sentence encoding models on a brain decoding task, where the sentence that an experimental participant has seen must be predicted from the fMRI signal evoked by the sentence. We take a pre-trained BERT architecture as a baseline sentence encoding model and fine-tune it on a variety of natural language understanding (NLU) tasks, asking which lead to improvements in brain-decoding performance. We find that none of the sentence encoding tasks tested yield si…
Beyond representational alignment with brain-guided language models for robust reasoning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（北京大学通用人工智能国家重点实验室、智能科学与技术学院）；Department of Psychological and Cognitive Sciences, Tsinghua University（清华大学心理与认知科学系）；Microsoft Research Asia（微软亚洲研究院）
Shared representations in brains and models reveal a two-route cortical organization during scene perception
Department of Cognition, Development and Education Psychology, Faculty of Psychology, University of Barcelona（认知、发展与教育心理学系，心理学学院，巴塞罗那大学）；Institute of Neurosciences, University of Barcelona（神经科学研究所，巴塞罗那大学）；Bellvitge Institute for Biomedical Research（Bellvitge生物医学研究 institute）
Platonic Representations in the Human Brain: Unsupervised Recovery of Universal Geometry
Department of Cognition, Development and Education Psychology, University of Barcelona（巴塞罗那大学认知、发展与教育心理学系）；Institute of Neurosciences, University of Barcelona（巴塞罗那大学神经科学研究所）；Department of Computer Science, Cornell University（康奈尔大学计算机科学系）；Bellvitge Institute for Biomedical Research, Spain（西班牙贝尔维德生物医学研究 institute）
Multivariate pattern analysis (MVPA) or brain decoding methods have become standard practice in analysing fMRI data. Although decoding methods have been extensively applied in Brain Computing Interfaces (BCI), these methods have only recently been applied to time-series neuroimaging data such as MEG and EEG to address experimental questions in Cognitive Neuroscience. In a tutorial-style review, we describe a broad set of options to inform future time-series decoding studies from a Cognitive Neuroscience perspective. Using example MEG data, we illustrate the effects that different options in th…
Perceptual similarity is a cognitive judgment that represents the end-stage of a complex cascade of hierarchical processing throughout visual cortex. Previous studies have shown a correspondence between the similarity of coarse-scale fMRI activation patterns and the perceived similarity of visual stimuli, suggesting that visual objects that appear similar also share similar underlying patterns of neural activation. Here we explore the temporal relationship between the human brain's time-varying representation of visual patterns and behavioral judgments of perceptual similarity. The visual stim…
