Naturalistic fMRI characterizes the brain as a dynamic predictive engine driven by continuous sensory streams. However, modeling the causal forward evolution in realistic neural simulation is impeded by the timescale mismatch between multimodal inputs and the complex topology of cortical networks. To address these challenges, we introduce BrainVista, a multimodal autoregressive framework designed to model the causal evolution of brain states. BrainVista incorporates Network-wise Tokenizers to disentangle system-specific dynamics and a Spatial Mixer Head that captures inter-network information flow without compromising functional boundaries. Furthermore, we propose a novel Stimulus-to-Brain (S2B) masking mechanism to synchronize high-frequency sensory stimuli with hemodynamically filtered signals, enabling strict, history-only causal conditioning. We validate our framework on Algonauts 2025, CineBrain, and HAD, achieving state-of-the-art fMRI encoding performance. In long-horizon rollout settings, our model yields substantial improvements over baselines, increasing pattern correlation by 36.0\% and 33.3\% on relative to the strongest baseline Algonauts 2025 and CineBrain, respectively.
University of Tokyo（东京大学）；University of Oxford（牛津大学）；ATR（国际电气通信基础技术研究所）；AIST（产业技术综合研究所）
Multi-regional interaction among neuronal populations underlies the brain's processing of rich sensory information in our daily lives. Recent neuroscience and neuroimaging studies have increasingly used naturalistic stimuli and experimental design to identify such realistic sensory computation in the brain. However, existing methods for cross-areal interaction analysis with dimensionality reduction, such as reduced-rank regression and canonical correlation analysis, have limited applicability and interpretability in naturalistic settings because they usually do not appropriately 'demix' neural…
Naturalistic fMRI has been suggested to be a powerful alternative for investigations of human brain function. Stimulus-induced activation has been playing an essential role in fMRI-based brain function analyses. Due to the complexity of the stimuli, however, detection of activations induced by naturalistic stimuli (AINSs) has been a tricky problem, as AINS cannot be detected simply in a model-driven way. In this study, we proposed a method to detect AINS in a pseudo model-driven way. Inspired by the strategy of utilizing the commonalities among the brains exposed to the same stimuli for inter-…
School of Computing and Engineering, University of West London（西伦敦大学计算与工程学院）
Objectives: Functional connectivity triggered by naturalistic stimulus (e.g., movies) and machine learning techniques provide a great insight in exploring the brain functions such as fluid intelligence. However, functional connectivity are considered to be multi-layered, while traditional machine learning based on individual models not only are limited in performance, but also fail to extract multi-dimensional and multi-layered information from brain network. Methods: In this study, inspired by multi-layer brain network structure, we propose a new method namely Weighted Ensemble-model and Netw…
Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Most ongoing efforts have focused on training decoders on specific, stereotyped tasks in laboratory settings. Implementing brain-computer interfaces (BCIs) in natural settings requires adaptive strategies and scalable algorithms that require minimal supervision. Here we propose an unsupervised approach to decoding neural states from human brain recordings acquired in a naturalistic context. We demonstrate our approach on continuous long-term electr…
Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation
South China Normal University（华南师范大学）；Center for Studies of Psychological Application, South China Normal University（华南师范大学心理应用研究中心）；School of Psychology, South China Normal University（华南师范大学心理学院）；Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University（华南师范大学广东省心理健康与认知科学重点实验室）
NeuroWorld: A Latent Brain World Model for Stimulus-Conditioned Human Brain Dynamics
National University of Singapore（新加坡国立大学）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院医学系）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）；Stanford University（斯坦福大学）；Department of Psychology, Stanford University（斯坦福大学心理学系）
A large dataset of human EEG responses to short naturalistic videos for studying dynamic visual event processing
Freie Universität Berlin（柏林自由大学）；Charité – Universitätsmedizin Berlin（柏林夏里特医科大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）；University of Amsterdam（阿姆斯特丹大学）
University of Washington（华盛顿大学）；Center for Neurotechnology（神经技术中心）；University of Washington Institute for Neuroengineering（华盛顿大学神经工程研究所）；eScience Institute, University of Washington（华盛顿大学电子科学研究所）
Recent technological advances in brain recording and artificial intelligence are propelling a new paradigm in neuroscience beyond the traditional controlled experiment. Rather than focusing on cued, repeated trials, naturalistic neuroscience studies neural processes underlying spontaneous behaviors performed in unconstrained settings. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term. Here we describe an automated approach for analyzing simultaneously recorded long-term, naturali…
Multimodal Recurrent Ensembles for Predicting Brain Responses to Naturalistic Movies (Algonauts 2025)
Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；TU Dresden（德累斯顿技术大学）；School for Embedded and Composite AI (SECAI)（嵌入式与复合人工智能学院）；Center for Scalable Data Analytics & AI (ScaDS.AI)（可扩展数据与人工智能中心）
Flatiron Institute（弗莱德哈钦森研究所）；Columbia University（哥伦比亚大学）；NYU Medical Center（纽约大学医学中心）
Learning latent features from time series data is an important problem in both machine learning and brain function. One approach, called Slow Feature Analysis (SFA), leverages the slowness of many salient features relative to the rapidly varying input signals. Furthermore, when trained on naturalistic stimuli, SFA reproduces interesting properties of cells in the primary visual cortex and hippocampus, suggesting that the brain uses temporal slowness as a computational principle for learning latent features. However, despite the potential relevance of SFA for modeling brain function, there is c…
Intrinsic Goals for Autonomous Agents: Model-Based Exploration in Virtual Zebrafish Predicts Ethological Behavior and Whole-Brain Dynamics
Neuroscience Institute, Carnegie Mellon University（卡内基梅隆大学神经科学研究所）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Machine Learning Department, Carnegie Mellon University（卡内基梅隆大学机器学习系）；IBM Thomas J. Watson Research Center, IBM Research（IBM沃森研究中心）
Previous research has shown a clear relationship between sleep and memory, examining the impact of sleep deprivation on key cognitive processes over very short durations or in special populations. Here, we show, in a longitudinal 16 week study, that naturalistic, unfettered sleep modulations in healthy adults have significant impacts on the brain. Using a dynamic networks approach combined with hierarchical statistical modelling, we show that the flexibility of particular brain regions that span a large network including regions in occipital, temporal, and frontal cortex increased when partici…
Generative adversarial networks (GANs) are recently highly successful in generative applications involving images and start being applied to time series data. Here we describe EEG-GAN as a framework to generate electroencephalographic (EEG) brain signals. We introduce a modification to the improved training of Wasserstein GANs to stabilize training and investigate a range of architectural choices critical for time series generation (most notably up- and down-sampling). For evaluation we consider and compare different metrics such as Inception score, Frechet inception distance and sliced Wasser…
Monash University Malaysia（莫纳什大学马来西亚分校）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University College London（伦敦大学学院）；University of Texas at Dallas（德克萨斯大学达拉斯分校）
We consider the challenges in extracting stimulus-related neural dynamics from other intrinsic processes and noise in naturalistic functional magnetic resonance imaging (fMRI). Most studies rely on inter-subject correlations (ISC) of low-level regional activity and neglect varying responses in individuals. We propose a novel, data-driven approach based on low-rank plus sparse (L+S) decomposition to isolate stimulus-driven dynamic changes in brain functional connectivity (FC) from the background noise, by exploiting shared network structure among subjects receiving the same naturalistic stimuli…
fMRI predictors based on language models of increasing complexity recover brain left lateralization
Centre d’Analyse et de Mathématique Sociales（社会分析与数学中心）；CNRS（法国国家科学研究中心）；EHESS（法国社会科学高等研究院）；Cognitive Neuroimaging Unit（认知神经成像单元）；INSERM（法国国家健康与医学研究院）；CEA（法国原子能委员会）；Neurospin center（Neurospin中心）
Music, being a multifaceted stimulus evolving at multiple timescales, modulates brain function in a manifold way that encompasses not only the distinct stages of auditory perception but also higher cognitive processes like memory and appraisal. Network theory is apparently a promising approach to describe the functional reorganization of brain oscillatory dynamics during music listening. However, the music induced changes have so far been examined within the functional boundaries of isolated brain rhythms. Using naturalistic music, we detected the functional segregation patterns associated wit…
Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining
Department of Biomedical Engineering, Southern University of Science and Technology, China（南方科技大学生物医学工程系，中国）；School of Biomedical Engineering, Shenzhen University, China（深圳大学生物医学工程学院，中国）
Flexibility and Invariance of Object Representations in the Human Brain
Neuroscience Institute, Carnegie Mellon University（卡内基梅隆大学神经科学研究所）；Department of Psychology, Carnegie Mellon University（卡内基梅隆大学心理学系）；Department of Machine Learning, Carnegie Mellon University（卡内基梅隆大学机器学习系）；Department of Neurosurgery, University of Rochester Medical Center（罗切斯特大学医学中心神经外科）
The shared response model provides a simple but effective framework to analyse fMRI data of subjects exposed to naturalistic stimuli. However when the number of subjects or runs is large, fitting the model requires a large amount of memory and computational power, which limits its use in practice. In this work, we introduce the FastSRM algorithm that relies on an intermediate atlas-based representation. It provides considerable speed-up in time and memory usage, hence it allows easy and fast large-scale analysis of naturalistic-stimulus fMRI data. Using four different datasets, we show that ou…
We propose an algorithm grounded in dynamical systems theory that generalizes manifold learning from a global state representation, to a network of local interacting manifolds termed a Generative Manifold Network (GMN). Manifolds are discovered using the convergent cross mapping (CCM) causal inference algorithm which are then compressed into a reduced redundancy network. The representation is a network of manifolds embedded from observational data where each orthogonal axis of a local manifold is an embedding of a individually identifiable neuron or brain area that has exact correspondence in …
Recurrent network dynamics explain the time course of perceptual grouping in natural scenes
Netherlands Institute for Neuroscience（荷兰神经科学研究所）；Brown University（布朗大学）；Carney Institute for Brain Science, Brown University（布朗大学卡尼脑科学研究所）；Centrum Wiskunde & Informatica（数学与计算机科学中心）；University of Amsterdam（阿姆斯特丹大学）；Sorbonne Université（索邦大学）；Institut National de la Santé et de la Recherche Médicale（国家健康与医学研究院）；Centre National de la Recherche Scientifique（法国国家科学研究中心）；Institut de la Vision（视觉研究所）；VU University（自由大学）；Academic University Medical Center（学术医学中心）
The University of Tokyo（东京大学）；Princeton University（普林斯顿大学）；Facebook Reality Labs（Facebook现实实验室）；Columbia University（哥伦比亚大学）；Intel Corporation（英特尔公司）
With the wide adoption of functional magnetic resonance imaging (fMRI) by cognitive neuroscience researchers, large volumes of brain imaging data have been accumulated in recent years. Aggregating these data to derive scientific insights often faces the challenge that fMRI data are high-dimensional, heterogeneous across people, and noisy. These challenges demand the development of computational tools that are tailored both for the neuroscience questions and for the properties of the data. We review a few recently developed algorithms in various domains of fMRI research: fMRI in naturalistic ta…
Rice University（莱斯大学）；University of Minnesota（明尼苏达大学）；Baylor College of Medicine（贝勒医学院）
Continuous control and planning remains a major challenge in robotics and machine learning. Neuroscience offers the possibility of learning from animal brains that implement highly successful controllers, but it is unclear how to relate an animal's behavior to control principles. Animals may not always act optimally from the perspective of an external observer, but may still act rationally: we hypothesize that animals choose actions with highest expected future subjective value according to their own internal model of the world. Their actions thus result from solving a different optimal contro…
Probabilistic Prediction of Neural Dynamics via Autoregressive Flow Matching
Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands（荷兰奈梅亨拉德堡德大学唐德斯脑、认知与行为研究所机器学习和神经计算系）；Department of Cognitive Neuroscience, Maastricht University, Maastricht, the Netherlands（荷兰马斯特里赫特大学认知神经科学系）；Maastricht Brain Imaging Center, Maastricht University, Maastricht, the Netherlands（荷兰马斯特里赫特大学马斯特里赫特脑成像中心）
