Current non-invasive neuroimaging techniques trade off between spatial resolution and temporal resolution. While magnetoencephalography (MEG) can capture rapid neural dynamics and functional magnetic resonance imaging (fMRI) can spatially localize brain activity, a unified picture that preserves both high resolutions remains an unsolved challenge with existing source localization or MEG-fMRI fusion methods, especially for single-trial naturalistic data. We collected whole-head MEG when subjects listened passively to more than seven hours of narrative stories, using the same stimuli in an open fMRI dataset (LeBel et al., 2023). We developed a transformer-based encoding model that combines the MEG and fMRI from these two naturalistic speech comprehension experiments to estimate latent cortical source responses with high spatiotemporal resolution. Our model is trained to predict MEG and fMRI from multiple subjects simultaneously, with a latent layer that represents our estimates of reconstructed cortical sources. Our model predicts MEG better than the common standard of single-modality encoding models, and it also yields source estimates with higher spatial and temporal fidelity than classic minimum-norm solutions in simulation experiments. We validated the estimated latent sources by showing its strong generalizability across unseen subjects and modalities. Estimated activity in our source space predict electrocorticography (ECoG) better than an ECoG-trained encoding model in an entirely new dataset. By integrating the power of large naturalistic experiments, MEG, fMRI, and encoding models, we propose a practical route towards millisecond-and-millimeter brain mapping.
Monash University Malaysia（莫纳什大学马来西亚分校）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University College London（伦敦大学学院）；University of Texas at Dallas（德克萨斯大学达拉斯分校）
We consider the challenges in extracting stimulus-related neural dynamics from other intrinsic processes and noise in naturalistic functional magnetic resonance imaging (fMRI). Most studies rely on inter-subject correlations (ISC) of low-level regional activity and neglect varying responses in individuals. We propose a novel, data-driven approach based on low-rank plus sparse (L+S) decomposition to isolate stimulus-driven dynamic changes in brain functional connectivity (FC) from the background noise, by exploiting shared network structure among subjects receiving the same naturalistic stimuli…
Naturalistic fMRI has been suggested to be a powerful alternative for investigations of human brain function. Stimulus-induced activation has been playing an essential role in fMRI-based brain function analyses. Due to the complexity of the stimuli, however, detection of activations induced by naturalistic stimuli (AINSs) has been a tricky problem, as AINS cannot be detected simply in a model-driven way. In this study, we proposed a method to detect AINS in a pseudo model-driven way. Inspired by the strategy of utilizing the commonalities among the brains exposed to the same stimuli for inter-…
The shared response model provides a simple but effective framework to analyse fMRI data of subjects exposed to naturalistic stimuli. However when the number of subjects or runs is large, fitting the model requires a large amount of memory and computational power, which limits its use in practice. In this work, we introduce the FastSRM algorithm that relies on an intermediate atlas-based representation. It provides considerable speed-up in time and memory usage, hence it allows easy and fast large-scale analysis of naturalistic-stimulus fMRI data. Using four different datasets, we show that ou…
Estimated connectomes by the means of neuroimaging techniques have enriched our knowledge of the organizational properties of the brain leading to the development of network-based clinical diagnostics. Unfortunately, to date, many of those network-based clinical diagnostics tools, based on the mere description of isolated instances of observed connectomes are noisy estimates of the true connectivity network. Modeling brain connectivity networks is therefore important to better explain the functional organization of the brain and allow inference of specific brain properties. In this report, we …
Cornell University（康奈尔大学）；Weill Cornell Medical College（威尔康奈尔医学院）
The increasing popularity of naturalistic paradigms in fMRI (such as movie watching) demands novel strategies for multi-subject data analysis, such as use of neural encoding models. In the present study, we propose a shared convolutional neural encoding method that accounts for individual-level differences. Our method leverages multi-subject data to improve the prediction of subject-specific responses evoked by visual or auditory stimuli. We showcase our approach on high-resolution 7T fMRI data from the Human Connectome Project movie-watching protocol and demonstrate significant improvement ov…
NeuroWorld: A Latent Brain World Model for Stimulus-Conditioned Human Brain Dynamics
National University of Singapore（新加坡国立大学）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院医学系）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）；Stanford University（斯坦福大学）；Department of Psychology, Stanford University（斯坦福大学心理学系）
The University of Tokyo（东京大学）；Princeton University（普林斯顿大学）；Facebook Reality Labs（Facebook现实实验室）；Columbia University（哥伦比亚大学）；Intel Corporation（英特尔公司）
With the wide adoption of functional magnetic resonance imaging (fMRI) by cognitive neuroscience researchers, large volumes of brain imaging data have been accumulated in recent years. Aggregating these data to derive scientific insights often faces the challenge that fMRI data are high-dimensional, heterogeneous across people, and noisy. These challenges demand the development of computational tools that are tailored both for the neuroscience questions and for the properties of the data. We review a few recently developed algorithms in various domains of fMRI research: fMRI in naturalistic ta…
In this paper I review the submission to the Explaining the Human Visual Brain Challenge 2019 in both the fMRI and MEG tracks. The goal was to construct neural network features which generate the so-called representational dissimilarity matrix (RDM) which is most similar to the one extracted from fMRI and MEG data upon viewing a set of images. I review exploring the optimal granularity of the receptive field, a construction of intermediate surrogate features using Multidimensional Scaling and modelling them using neural network features. I also point out some peculiarities of the RDM construct…
Directorate of Technology and Innovation（技术与创新局）；Chamber of Deputies（众议院）
Deep predictive coding networks are neuroscience-inspired unsupervised learning models that learn to predict future sensory states. We build upon the PredNet implementation by Lotter, Kreiman, and Cox (2016) to investigate if predictive coding representations are useful to predict brain activity in the visual cortex. We use representational similarity analysis (RSA) to compare PredNet representations to functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data from the Algonauts Project. In contrast to previous findings in the literature (Khaligh-Razavi &Kriegeskorte, …
Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining
Department of Biomedical Engineering, Southern University of Science and Technology, China（南方科技大学生物医学工程系，中国）；School of Biomedical Engineering, Shenzhen University, China（深圳大学生物医学工程学院，中国）
Multimodal Recurrent Ensembles for Predicting Brain Responses to Naturalistic Movies (Algonauts 2025)
Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；TU Dresden（德累斯顿技术大学）；School for Embedded and Composite AI (SECAI)（嵌入式与复合人工智能学院）；Center for Scalable Data Analytics & AI (ScaDS.AI)（可扩展数据与人工智能中心）
Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation
South China Normal University（华南师范大学）；Center for Studies of Psychological Application, South China Normal University（华南师范大学心理应用研究中心）；School of Psychology, South China Normal University（华南师范大学心理学院）；Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University（华南师范大学广东省心理健康与认知科学重点实验室）
A large dataset of human EEG responses to short naturalistic videos for studying dynamic visual event processing
Freie Universität Berlin（柏林自由大学）；Charité – Universitätsmedizin Berlin（柏林夏里特医科大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）；University of Amsterdam（阿姆斯特丹大学）
School of Computing and Engineering, University of West London（西伦敦大学计算与工程学院）
Objectives: Functional connectivity triggered by naturalistic stimulus (e.g., movies) and machine learning techniques provide a great insight in exploring the brain functions such as fluid intelligence. However, functional connectivity are considered to be multi-layered, while traditional machine learning based on individual models not only are limited in performance, but also fail to extract multi-dimensional and multi-layered information from brain network. Methods: In this study, inspired by multi-layer brain network structure, we propose a new method namely Weighted Ensemble-model and Netw…
fMRI predictors based on language models of increasing complexity recover brain left lateralization
Centre d’Analyse et de Mathématique Sociales（社会分析与数学中心）；CNRS（法国国家科学研究中心）；EHESS（法国社会科学高等研究院）；Cognitive Neuroimaging Unit（认知神经成像单元）；INSERM（法国国家健康与医学研究院）；CEA（法国原子能委员会）；Neurospin center（Neurospin中心）
Neuronal Correlates of Semantic Event Classes during Presentation of Complex Naturalistic Stimuli: Anatomical Patterns, Context-Sensitivity, and Potential Impact on shared Human-Robot Ontologies
Progress in natural language processing (NLP) models that estimate representations of word sequences has recently been leveraged to improve the understanding of language processing in the brain. However, these models have not been specifically designed to capture the way the brain represents language meaning. We hypothesize that fine-tuning these models to predict recordings of brain activity of people reading text will lead to representations that encode more brain-activity-relevant language information. We demonstrate that a version of BERT, a recently introduced and powerful language model,…
The advent of deep learning has a profound effect on visual neuroscience. It paved the way for new models to predict neural data. Although deep convolutional neural networks are explicitly trained for categorization, they learn a representation similar to a biological visual system. But categorization is not the only goal of the human visual system. Hence, the representation of a classification algorithm may not completely explain the visual processing stages. Here, I modified the traditional Siamese network loss function (Contrastive loss) to train them directly on neural dissimilarity. This …
Magnetoencephalography and electroencephalography (M/EEG) are non-invasive modalities that measure the weak electromagnetic fields generated by neural activity. Estimating the location and magnitude of the current sources that generated these electromagnetic fields is a challenging ill-posed regression problem known as \emph{source imaging}. When considering a group study, a common approach consists in carrying out the regression tasks independently for each subject. An alternative is to jointly localize sources for all subjects taken together, while enforcing some similarity between them. By …
Université de Montréal（蒙特利尔大学）；Mila – Quebec Artificial Intelligence Institute（米拉-魁北克人工智能研究所）；IMT Atlantique（IMT大西洋国立高等矿业电信学校）；IVADO, Université de Montréal（蒙特利尔大学IVADO研究所）；Hôpital du Sacré-Cœur de Montréal, Santé Québec Nord-de-l’Île-de-Montréal - Universitaire（蒙特利尔圣心医院，魁北克健康部蒙特利尔岛北-大学网络）；Faculty of Medicine, Université de Montréal（蒙特利尔大学医学院）
We study source localization from high dimensional M/EEG data by extending a multiscale method based on Entropic inference devised to increase the spatial resolution of inverse problems. This method is used to construct informative prior distributions in a manner inspired in the context of fMRI (Amaral et al 2004). We construct a set of renormalized lattices that approximate the cortex region where the source activity is located and address the related problem of defining the relevant variables in a coarser scale representation of the cortex. The priors can be used in conjunction with other Ba…
Perceptual similarity is a cognitive judgment that represents the end-stage of a complex cascade of hierarchical processing throughout visual cortex. Previous studies have shown a correspondence between the similarity of coarse-scale fMRI activation patterns and the perceived similarity of visual stimuli, suggesting that visual objects that appear similar also share similar underlying patterns of neural activation. Here we explore the temporal relationship between the human brain's time-varying representation of visual patterns and behavioral judgments of perceptual similarity. The visual stim…
Mapping minds not averages: a scalable subject-specific manifold learning framework for neuroimaging data
Electrical and Computer Engineering, Georgia Institute of Technology（电子与计算机工程系，佐治亚理工学院）；(TReNDS) Translational Research in Neuroimaging & Data Science center, Georgia State University, Georgia Institute of Technology, & Emory University（(TReNDS) 神经影像与数据科学转化研究中心，佐治亚州立大学，佐治亚理工学院，及埃默里大学）
Independent-Component-Based Encoding Models of Brain Activity During Story Comprehension
School of Electrical and Computer Engineering, Georgia Institute of Technology（佐治亚理工学院电子与计算机工程学院）；School of Psychological and Brain Sciences, Georgia Institute of Technology（佐治亚理工学院心理学与脑科学学院）；Department of Linguistics, Stanford University（斯坦福大学语言学系）
Functional MRI (fMRI) has been widely used to study activity patterns in the human brain. It infers neuronal activity from the associated hemodynamic response, which fundamentally limits its spatial and temporal specificity. In mice, the Direct Imaging of Neuronal Activity (DIANA) method revealed MRI signals that correlated with intracellular voltage changes, showing high spatial and temporal specificity. In this work we attempted DIANA in humans. Four visual paradigms were tested, exploring different stimulus types (flickering noise patterns, and naturalistic images) and stimulus durations (5…
Conventional neuroimaging analyses have revealed the computational specificity of localized brain regions, exploiting the power of the subtraction technique in fMRI and event-related potential analyses in EEG. Moving beyond this convention, many researchers have begun exploring network-based neurodynamics and coordination between brain regions as a function of behavioral parameters or environmental statistics; however, most approaches average evoked activity across the experimental session to study task-dependent networks. Here, we examined on-going oscillatory activity and use a methodology t…
Princeton University（普林斯顿大学）；Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；Carnegie Mellon University（卡内基梅隆大学）；Google Research（谷歌研究院）
To study information processing in the brain, neuroscientists manipulate experimental stimuli while recording participant brain activity. They can then use encoding models to find out which brain "zone" (e.g. which region of interest, volume pixel or electrophysiology sensor) is predicted from the stimulus properties. Given the assumptions underlying this setup, when stimulus properties are predictive of the activity in a zone, these properties are understood to cause activity in that zone. In recent years, researchers have used neural networks to construct representations that capture the div…
Peking University（北京大学）；Meta Inc.（元宇宙公司）；Beijing Normal University（北京师范大学）
Neural decoding is still a challenge and hot topic in neurocomputing science. Recently, many studies have shown that brain network patterns containing rich spatial and temporal structure information, which represents the activation information of brain under external stimuli. %Therefore, the research of decoding stimuli from brain network received extensive more attention. The traditional method extracts brain network features directly from the common machine learning method, then puts these features into the classifier, and realizes to decode external stimuli. However, this method cannot effe…
Coherence in the brain unfolds across separable temporal regimes
Department of Adult Psychiatry and Psychotherapy, University of Zurich, Zurich, Switzerland（大学精神病学与心理学系，苏黎世大学，苏黎世，瑞士）；Department of Translation and Language Sciences, University Pompeu Fabra, Barcelona, Spain（翻译与语言科学系，庞培法拉大学，巴塞罗那，西班牙）；Department of Neuroradiology, Clinical Neuroscience Center, University Hospital Zurich, Zurich, Switzerland（神经放射学系，临床神经科学中心，苏黎世大学医院，苏黎世，瑞士）；Department of Psychology, University of Oslo, Oslo, Norway（心理学系，奥斯陆大学，奥斯陆，挪威）；Cognitive and Affective Neuroscience Unit, University of Zurich, Zurich, Switzerland（认知与情感神经科学单元，苏黎世大学，苏黎世，瑞士）；Center for Clinical Neuroscience and Cognition, University of Groningen, Groningen, Netherlands（临床神经科学与认知中心，格罗宁根大学，格罗宁根，荷兰）；Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland（苏黎世神经科学中心，苏黎世大学和ETH苏黎世分校，苏黎世，瑞士）
