The brain-AI convergence: Predictive and generative world models for general-purpose computation
Chinese Institute for Brain Research (CIBR)（北京脑科学与类脑研究中心）
Shogo Ohmae, Keiko Ohmae
备注：
22 pages, 4 figures. Related to our earlier preprint "The brain versus AI" (arXiv:2411.16075) but a distinct article. The earlier work surveyed broad brain-AI parallels; here we focus on world-model-based computation and convergent evolution between the brain and AI, especially large language models
Recent advances in general-purpose AI systems with attention-based transformers offer a potential window into how the neocortex and cerebellum, despite their relatively uniform circuit architectures, give rise to diverse functions and, ultimately, to human intelligence. This Perspective provides a cross-domain comparison between the brain and AI that goes beyond the traditional focus on visual processing, adopting the emerging perspecive of world-model-based computation. Here, we identify shared computational mechanisms in the attention-based neocortex and the non-attentional cerebellum: both predict future world events from past inputs and construct internal world models through prediction-error learning. These predictive world models are repurposed for seemingly distinct functions -- understanding in sensory processing and generation in motor processing -- enabling the brain to achieve multi-domain capabilities and human-like adaptive intelligence. Notably, attention-based AI has independently converged on a similar learning paradigm and world-model-based computation. We conclude that these shared mechanisms in both biological and artificial systems constitute a core computational foundation for realizing diverse functions including high-level intelligence, despite their relatively uniform circuit structures. Our theoretical insights bridge neuroscience and AI, advancing our understanding of the computational essence of intelligence.
Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken s…
Bayesian interpretations of neural processing require that biological mechanisms represent and operate upon probability distributions in accordance with Bayes' theorem. Many have speculated that synaptic failure constitutes a mechanism of variational, i.e., approximate, Bayesian inference in the brain. Whereas models have previously used synaptic failure to sample over uncertainty in model parameters, we demonstrate that by adapting transmission probabilities to learned network weights, synaptic failure can sample not only over model uncertainty, but complete posterior predictive distributions…
University of Edinburgh（爱丁堡大学）；University of Sussex（萨塞克斯大学）
Predictive coding offers a potentially unifying account of cortical function -- postulating that the core function of the brain is to minimize prediction errors with respect to a generative model of the world. The theory is closely related to the Bayesian brain framework and, over the last two decades, has gained substantial influence in the fields of theoretical and cognitive neuroscience. A large body of research has arisen based on both empirically testing improved and extended theoretical and mathematical models of predictive coding, as well as in evaluating their potential biological plau…
Directorate of Technology and Innovation（技术与创新局）；Chamber of Deputies（众议院）
Deep predictive coding networks are neuroscience-inspired unsupervised learning models that learn to predict future sensory states. We build upon the PredNet implementation by Lotter, Kreiman, and Cox (2016) to investigate if predictive coding representations are useful to predict brain activity in the visual cortex. We use representational similarity analysis (RSA) to compare PredNet representations to functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data from the Algonauts Project. In contrast to previous findings in the literature (Khaligh-Razavi &Kriegeskorte, …
Targeted stimulation of the brain has the potential to treat mental illnesses. We propose an approach to help design the stimulation protocol by identifying electrical dynamics across many brain regions that relate to illness states. We model multi-region electrical activity as a superposition of activity from latent networks, where the weights on the latent networks relate to an outcome of interest. In order to improve on drawbacks of latent factor modeling in this context, we focus on supervised autoencoders (SAEs), which can improve predictive performance while maintaining a generative mode…
Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement
Zuckerman Mind Brain Behavior Institute, Columbia University（扎克曼脑行为研究所，哥伦比亚大学）；Department of Electrical Engineering, Columbia University（电气工程系，哥伦比亚大学）；Department of Computer Science, Columbia University（计算机科学系，哥伦比亚大学）
In this study we adopt predictive modelling to identify simultaneously commonalities and differences in multi-modal brain networks acquired within subjects. Typically, predictive modelling of functional connectomes from structural connectomes explores commonalities across multimodal imaging data. However, direct application of multivariate approaches such as sparse Canonical Correlation Analysis (sCCA) applies on the vectorised elements of functional connectivity across subjects and it does not guarantee that the predicted models of functional connectivity are Symmetric Positive Matrices (SPD)…
CerCo CNRS（法国国家科学研究中心大脑与认知研究中心）；Université de Toulouse（图卢兹大学）；IRIT CNRS（法国国家科学研究中心图卢兹计算机科学研究所）；ANITI（图卢兹人工智能跨学科研究所）
Deep neural networks excel at image classification, but their performance is far less robust to input perturbations than human perception. In this work we explore whether this shortcoming may be partly addressed by incorporating brain-inspired recurrent dynamics in deep convolutional networks. We take inspiration from a popular framework in neuroscience: 'predictive coding'. At each layer of the hierarchical model, generative feedback 'predicts' (i.e., reconstructs) the pattern of activity in the previous layer. The reconstruction errors are used to iteratively update the network's representat…
University of Cambridge（剑桥大学）；University of Rome “Tor Vergata”（罗马大学“Tor Vergata”）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）
The characterisation of the brain as a "connectome", in which the connections are represented by correlational values across timeseries and as summary measures derived from graph theory analyses, has been very popular in the last years. However, although this representation has advanced our understanding of the brain function, it may represent an oversimplified model. This is because the typical fMRI datasets are constituted by complex and highly heterogeneous timeseries that vary across space (i.e., location of brain regions). We compare various modelling techniques from deep learning and geo…
Why the Brain Consolidates: Predictive Forgetting for Optimal Generalisation
Huawei Noah’s Ark Lab（华为诺亚实验室）；AI Centre, Department of Computer Science, University College London（大学计算机科学系人工智能中心）；UCL Institute of Cognitive Neuroscience, University College London（大学认知神经科学研究所）；UCL Queen Square Institute of Neurology, University College London（大学Queen Square神经学研究所）
The use of machine learning (ML) algorithms has significantly increased in neuroscience. However, from the vast extent of possible ML algorithms, which one is the optimal model to predict the target variable? What are the hyperparameters for such a model? Given the plethora of possible answers to these questions, in the last years, automated machine learning (autoML) has been gaining attention. Here, we apply an autoML library called TPOT which uses a tree-based representation of machine learning pipelines and conducts a genetic-programming based approach to find the model and its hyperparamet…
Meta predictive learning model of languages in neural circuits
Sun Yat-sen University（中山大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Guangdong Provincial Key Laboratory of Magnetoelectric Physics and Devices（广东省磁电物理与器件重点实验室）
Deep brain stimulation(DBS)has become an effective intervention for advanced Parkinson's disease, but the exact mechanism of DBS is still unclear. In this review, we discuss the history of DBS, the anatomy and internal architecture of the basal ganglia(BG), the abnormal pathological changes of the BG in Parkinson's disease, and how computational models can help understand and advance DBS. We also describe two types of models:mathematical theoretical models and clinical predictive models. Mathematical theoretical models simulate neurons or neural networks of BG to shed light on the mechanistic …
Reassessing prediction in the brain: Pre-onset neural encoding during natural listening does not reflect pre-activation
Donders Institute for Brain, Cognition, and Behaviour（达恩斯大脑、认知与行为研究所）；Radboud University（拉德堡德大学）；Radboud University Medical Center（拉德堡德大学医学中心）；Jagiellonian University（雅盖隆大学）
Syntactic parsing is the task of assigning a syntactic structure to a sentence. There are two popular syntactic parsing methods: constituency and dependency parsing. Recent works have used syntactic embeddings based on constituency trees, incremental top-down parsing, and other word syntactic features for brain activity prediction given the text stimuli to study how the syntax structure is represented in the brain's language network. However, the effectiveness of dependency parse trees or the relative predictive power of the various syntax parsers across brain areas, especially for the listeni…
The free energy principle has been proposed as a unifying account of brain function. It is closely related, and in some cases subsumes, earlier unifying ideas such as Bayesian inference, predictive coding, and active learning. This article clarifies these connections, teasing apart distinctive and shared predictions.
Decoding Predictive Inference in Visual Language Processing via Spatiotemporal Neural Coherence
Department of Physics（物理系）；Carnegie-Mellon University（卡内基梅隆大学）；Department of Linguistics（语言学系）；University of Salzburg（萨尔茨堡大学）；Purdue University（普渡大学）；University of Alabama（阿拉巴马大学）；Department of Communicative Disorders（沟通障碍系）
The shape of the brain's connections is predictive of cognitive performance: an explainable machine learning study
Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根妇女医院）；Massachusetts General Hospital（麻省总医院）；The University of Sydney（悉尼大学）；Beijing Institute of Technology（北京理工大学）；Boston University（波士顿大学）；University of Electronic Science and Technology of China（电子科技大学）；Harvard University（哈佛大学）；Harvard-MIT Health Sciences and Technology（哈佛-麻省理工健康科学与技术学部）
Emerging evidence showed that major depressive disorder (MDD) is associated with disruptions of brain structural and functional networks, rather than impairment of isolated brain region. Thus, connectome-based models capable of predicting the depression severity at the individual level can be clinically useful. Here, we applied a machine-learning approach to predict the severity of depression using resting-state networks derived from source-reconstructed Electroencephalography (EEG) signals. Using regression models and three independent EEG datasets (N=328), we tested whether resting state fun…
Humans constantly interact with digital devices that disregard their feelings. However, the synergy between human and technology can be strengthened if the technology is able to distinguish and react to human emotions. Models that rely on unconscious indications of human emotions, such as (neuro)physiological signals, hold promise in personalization of feedback and adaptation of the interaction. The current study elaborated on adopting a predictive approach in studying human emotional processing based on brain activity. More specifically, we investigated the proposition of predicting self-repo…
Modeling brain dynamics to better understand and control complex behaviors underlying various cognitive brain functions are of interests to engineers, mathematicians, and physicists from the last several decades. With a motivation of developing computationally efficient models of brain dynamics to use in designing control-theoretic neurostimulation strategies, we have developed a novel data-driven approach in a long short-term memory (LSTM) neural network architecture to predict the temporal dynamics of complex systems over an extended long time-horizon in future. In contrast to recent LSTM-ba…
Tri-Institutional Center for Translational Research in Neuroimaging and Data Science（三机构神经影像与数据科学转化研究中心）；Georgia State（佐治亚州立大学）；Georgia Tech（佐治亚理工学院）；Emory（埃默里大学）；Borealis AI（Borealis人工智能公司）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Apple Machine Learning Research（苹果机器学习研究院）；Université de Montréal（蒙特利尔大学）
Recent neuroimaging studies that focus on predicting brain disorders via modern machine learning approaches commonly include a single modality and rely on supervised over-parameterized models.However, a single modality provides only a limited view of the highly complex brain. Critically, supervised models in clinical settings lack accurate diagnostic labels for training. Coarse labels do not capture the long-tailed spectrum of brain disorder phenotypes, which leads to a loss of generalizability of the model that makes them less useful in diagnostic settings. This work presents a novel multi-sc…
Anatomical connectivity imposes strong constraints on brain function, but there is no general agreement about principles that govern its organization. Based on extensive quantitative data we tested the power of three models to predict connections of the primate cerebral cortex: architectonic similarity (structural model), spatial proximity (distance model) and thickness similarity (thickness model). Architectonic similarity showed the strongest and most consistent influence on connection features. This parameter was strongly associated with the presence or absence of inter-areal connections an…
Individual differences in human intelligence can be modeled and predicted from in vivo neurobiological connectivity. Many established modeling frameworks for predicting intelligence, however, discard higher-order information about individual differences in brain network topology, and show only moderate performance when generalized to make predictions in out-of-sample subjects. In this paper, we propose that connectome-based predictive modeling, a common predictive modeling framework for neuroscience data, can be productively modified to incorporate information about brain network topology and …
University Hospital Erlangen（埃尔兰根大学医院）；University Erlangen-Nürnberg（埃尔兰根-纽伦堡大学）；Newcastle University（纽卡斯尔大学）；York University（约克大学）；University of California Irvine（加州大学欧文分校）；University College London（伦敦大学学院）
How is information processed in the brain during perception? Mechanistic insight is achieved only when experiments are employed to test formal or computational models. In analogy to lesion studies, phantom perception may serve as a vehicle to understand the fundamental processing principles underlying auditory perception. With a special focus on tinnitus -- as the prime example of auditory phantom perception -- we review recent work at the intersection of artificial intelligence, psychology, and neuroscience. In particular, we discuss why everyone with tinnitus suffers from hearing loss, but n…
BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics
Department of Biomedical Engineering, Southern University of Science and Technology（南方科技大学生物医学工程系）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）
Generative causal testing to bridge data-driven models and scientific theories in language neuroscience
Computer Science Department, University of Texas at Austin（德克萨斯大学计算机科学系）；Microsoft Research（微软研究院）；Neurosurgery Department, University of California（加州大学神经外科系）；EECS Department, University of California（加州大学电子工程与计算机科学系）；Statistics Department, University of California（加州大学统计学系）；Center for Computational Biology, University of California（加州大学计算生物学中心）；Neuroscience Department, University of California（加州大学神经科学系）
Centre for Computational Intelligence, University of Essex（埃塞克斯大学计算智能中心）
The last decades have seen significant advancements in non-invasive neuroimaging technologies that have been increasingly adopted to examine human brain development. However, these improvements have not necessarily been followed by more sophisticated data analysis measures that are able to explain the mechanisms underlying functional brain development. For example, the shift from univariate (single area in the brain) to multivariate (multiple areas in brain) analysis paradigms is of significance as it allows investigations into the interactions between different brain regions. However, despite…
This tutorial paper refers to the use of graph-theoretic concepts for analyzing brain signals. For didactic purposes it splits into two parts: theory and application. In the first part, we commence by introducing some basic elements from graph theory and stemming algorithmic tools, which can be employed for data-analytic purposes. Next, we describe how these concepts are adapted for handling evolving connectivity and gaining insights into network reorganization. Finally, the notion of signals residing on a given graph is introduced and elements from the emerging field of graph signal processin…
Cognitive neuroscience explores the mechanisms of cognition by studying its structural and functional brain correlates. Here, we report the first systematic review that assesses how information from structural and functional neuroimaging methods can be integrated to investigate the brain substrates of cognition. Web of Science and Scopus databases were searched for studies of healthy young adult populations that collected cognitive data, and structural and functional neuroimaging data. Five percent of screened studies met all inclusion criteria. Next, 54% of included studies related cognitive …
Almost all statistical and machine learning methods in analyzing brain networks rely on distances and loss functions, which are mostly Euclidean or matrix norms. The Euclidean or matrix distances may fail to capture underlying subtle topological differences in brain networks. Further, Euclidean distances are sensitive to outliers. A few extreme edge weights may severely affect the distance. Thus it is necessary to use distances and loss functions that recognize topology of data. In this review paper, we survey various topological distance and loss functions from topological data analysis (TDA)…
Uniquely human abilities may arise from special-purpose brain circuitry, or from concerted general capacity increases due to our outsized brains. We forward a novel hypothesis of the relation between computational capacity and brain size, linking mathematical formalisms of grammars with the allometric increases in cortical-subcortical ratios that arise in large brains. In sum, i) thalamocortical loops compute formal grammars; ii) successive cortical regions describe grammar rewrite rules of increasing size; iii) cortical-subcortical ratios determine the quantity of stacks in single-stack pushd…
The idea that the brain functions so as to minimize certain costs pervades theoretical neuroscience. Since a cost function by itself does not predict how the brain finds its minima, additional assumptions about the optimization method need to be made to predict the dynamics of physiological quantities. In this context, steepest descent (also called gradient descent) is often suggested as an algorithmic principle of optimization potentially implemented by the brain. In practice, researchers often consider the vector of partial derivatives as the gradient. However, the definition of the gradient…
The last decades saw dramatic progress in brain research. These advances were often buttressed by probing single variables to make circumscribed discoveries, typically through null hypothesis significance testing. New ways for generating massive data fueled tension between the traditional methodology, used to infer statistically relevant effects in carefully-chosen variables, and pattern-learning algorithms, used to identify predictive signatures by searching through abundant information. In this article, we detail the antagonistic philosophies behind two quantitative approaches: certifying ro…
