Can neural networks learn goal-directed behaviour using similar strategies to the brain, by combining the relationships between the current state of the organism and the consequences of future actions? Recent work has shown that recurrent neural networks trained on goal based tasks can develop representations resembling those found in the brain, entorhinal cortex grid cells, for instance. Here we explore the evolution of the dynamics of their internal representations and compare this with experimental data. We observe that once a recurrent network is trained to learn the structure of its environment solely based on sensory prediction, an attractor based landscape forms in the network's representation, which parallels hippocampal place cells in structure and function. Next, we extend the predictive objective to include Q-learning for a reward task, where rewarding actions are dependent on delayed cue modulation. Mirroring experimental findings in hippocampus recordings in rodents performing the same task, this training paradigm causes nonlocal neural activity to sweep forward in space at decision points, anticipating the future path to a rewarded location. Moreover, prevalent choice and cue-selective neurons form in this network, again recapitulating experimental findings. Together, these results indicate that combining predictive, unsupervised learning of the structure of an environment with reinforcement learning can help understand the formation of hippocampus-like representations containing both spatial and task-relevant information.
Rodent hippocampal population codes represent important spatial information about the environment during navigation. Several computational methods have been developed to uncover the neural representation of spatial topology embedded in rodent hippocampal ensemble spike activity. Here we extend our previous work and propose a nonparametric Bayesian approach to infer rat hippocampal population codes during spatial navigation. To tackle the model selection problem, we leverage a nonparametric Bayesian model. Specifically, to analyze rat hippocampal ensemble spiking activity, we apply a hierarchic…
The discovery of place cells and other spatially modulated neurons in the hippocampal complex of rodents has been crucial to elucidating the neural basis of spatial cognition. More recently, the replay of neural sequences encoding previously experienced trajectories has been observed during consummatory behavior potentially with implications for quick memory consolidation and behavioral planning. Several promising models for robotic navigation and reinforcement learning have been proposed based on these and previous findings. Most of these models, however, use carefully engineered neural netwo…
The hippocampus and the striatum support episodic and procedural memory, respectively, and "place" and "response" learning within spatial navigation. Recently this dichotomy has been linked to "model-based" and "model-free" reinforcement learning. Here we present a well-constrained neural model of how both systems support spatial navigation, and apply the same model to more abstract problems such as sequential decision making. In particular, we show that if a task can be transformed into a Markov Decision Process, the machinery provided by the hippocampus and striatum can be utilized to solve …
University of Electronic Science and Technology of China（电子科技大学）
Recent experimental observations have shown that the reactivation of hippocampal place cells (PC) during sleep or immobility depicts trajectories that can go around barriers and can flexibly adapt to a changing maze layout. Such layout-conforming replay sheds a light on how the activity of place cells supports the learning of flexible navigation of an animal in a dynamically changing maze. However, existing computational models of replay fall short of generating layout-conforming replay, restricting their usage to simple environments, like linear tracks or open fields. In this paper, we propos…
It is now widely accepted that one of the roles of the hippocampus is to maintain episodic spatial representations, while parallel striatal pathways contribute to both declarative and procedural value computations by encoding different input-specific outcome predictions. In this paper we investigate the use of these brain mechanisms for action selection, linking them to model-based and model-free controllers for decision making. To this aim we propose a biologically inspired computational model that embodies these theories and explains the functioning of the hippocampal-striatal circuit in a r…
Peking University（北京大学）；IDG/McGovern Institute for Brain Research（IDG/麦戈文脑科学研究所）；Peking-Tsinghua Center for Life Sciences（北大-清华生命科学中心）；Academy for Advanced Interdisciplinary Studies, Peking University（北京大学前沿交叉学科研究院）；Center of Quantitative Biology, Peking University（北京大学定量生物学中心）
The spiking activity of the hippocampal place cells plays a key role in producing and sustaining an internalized representation of the ambient space---a cognitive map. These cells do not only exhibit location-specific spiking during navigation, but also may rapidly replay the navigated routs through endogenous dynamics of the hippocampal network. Physiologically, such reactivations are viewed as manifestations of "memory replays" that help to learn new information and to consolidate previously acquired memories by reinforcing synapses in the parahippocampal networks. Below we propose a computa…
Self-localization during navigation with noisy sensors in an ambiguous world is computationally challenging, yet animals and humans excel at it. In robotics, Simultaneous Location and Mapping (SLAM) algorithms solve this problem though joint sequential probabilistic inference of their own coordinates and those of external spatial landmarks. We generate the first neural solution to the SLAM problem by training recurrent LSTM networks to perform a set of hard 2D navigation tasks that include generalization to completely novel trajectories and environments. The hidden unit representations exhibit…
The Whole Brain Architecture Initiative（全脑架构倡议组织）；The University of Tokyo（东京大学）；RIKEN（理化学研究所）
In building artificial intelligence (AI) agents, referring to how brains function in real environments can accelerate development by reducing the design space. In this study, we propose a probabilistic generative model (PGM) for navigation in uncertain environments by integrating the neuroscientific knowledge of hippocampal formation (HF) and the engineering knowledge in robotics and AI, namely, simultaneous localization and mapping (SLAM). We follow the approach of brain reference architecture (BRA) (Yamakawa, 2021) to compose the PGM and outline how to verify the model. To this end, we surve…
Place cells in the rat hippocampus play a key role in creating the animal's internal representation of the world. During active navigation, these cells spike only in discrete locations, together encoding a map of the environment. Electrophysiological recordings have shown that the animal can revisit this map mentally, during both sleep and awake states, reactivating the place cells that fired during its exploration in the same sequence they were originally activated. Although consistency of place cell activity during active navigation is arguably enforced by sensory and proprioceptive inputs, …
We present a novel artificial cognitive mapping system using generative deep neural networks, called variational autoencoder/generative adversarial network (VAE/GAN), which can map input images to latent vectors and generate temporal sequences internally. The results show that the distance of the predicted image is reflected in the distance of the corresponding latent vector after training. This indicates that the latent space is self-organized to reflect the proximity structure of the dataset and may provide a mechanism through which many aspects of cognition are spatially represented. The pr…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；University of British Columbia（不列颠哥伦比亚大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Hippocampal place cells can encode spatial locations of an animal in physical or task-relevant spaces. We simulated place cell populations that encoded either Euclidean- or graph-based positions of a rat navigating to goal nodes in a maze with a graph topology, and used manifold learning methods such as UMAP and Autoencoders (AE) to analyze these neural population activities. The structure of the latent spaces learned by the AE reflects their true geometric structure, while PCA fails to do so and UMAP is less robust to noise. Our results support future applications of AE architectures to decip…
The grid firing patterns are thought to provide an efficient intrinsic metric capable of supporting universal spatial metric for mammalian spatial navigation in all environments. However, whether spatial representations of grid cells in the entorhinal cortex are determined by local environment cues or form globally coherent patterns remains undetermined. To explore this underlying mechanism, here we proposed a possible theoretical explanation to describe connection between the neural space and the physical environment and transition from a local anchored to a global coherent representation acc…
University Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；University Hospital Erlangen（埃尔朗根大学医院）
How does the mind organize thoughts? The hippocampal-entorhinal complex is thought to support domain-general representation and processing of structural knowledge of arbitrary state, feature and concept spaces. In particular, it enables the formation of cognitive maps, and navigation on these maps, thereby broadly contributing to cognition. It has been proposed that the concept of multi-scale successor representations provides an explanation of the underlying computations performed by place and grid cells. Here, we present a neural network based approach to learn such representations, and its …
Emergent Generalization by Representation Learning in Artificial Neural Networks
I-X Centre for AI in Science, Imperial College London（AI科学中心，帝国理工学院伦敦分校）；Department of Electrical and Electronic Engineering, Imperial College London（电气与电子工程系，帝国理工学院伦敦分校）
Brain-computer interface (BCI) systems have potential as assistive technologies for individuals with severe motor impairments. Nevertheless, individuals must first participate in many training sessions to obtain adequate data for optimizing the classification algorithm and subsequently acquiring brain-based control. Such traditional training paradigms have been dubbed unengaging and unmotivating for users. In recent years, it has been shown that the synergy of virtual reality (VR) and a BCI can lead to increased user engagement. This study created a 3-class BCI with a rather elaborate EEG sign…
