How does the primate brain combine generative and discriminative computations in vision?
Zuckerman Mind Brain Behavior Institute, Columbia University（哥伦比亚大学扎克曼脑与行为研究所）；School of Psychology & Neuroscience, University of Glasgow（格拉斯哥大学心理与神经科学学院）；Department of Brain and Cognitive Sciences, MIT（麻省理工学院脑与认知科学系）；McGovern Institute for Brain Research, MIT（麻省理工学院麦戈文脑研究所）；Quest for Intelligence, Schwarzman College of Computing, MIT（麻省理工学院施瓦茨曼计算学院智能探索中心）；Department of Psychology, New York University（纽约大学心理学系）；Brain and Cognitive Sciences, University of Rochester（罗切斯特大学脑与认知科学系）；Center for Visual Science, University of Rochester（罗切斯特大学视觉科学中心）；Department of Cognitive Science, Johns Hopkins University（约翰霍普金斯大学认知科学系）；Computer Science and Artificial Intelligence Laboratory, MIT（麻省理工学院计算机科学与人工智能实验室）；Department of Psychology, Harvard University（哈佛大学心理学系）；Center for Brain Science, Harvard University（哈佛大学脑科学中心）；Kempner Institute for Natural and Artificial Intelligence, Harvard University（哈佛大学肯普ner自然与人工智能研究所）；Department of Neuroscience, University of Minnesota（明尼苏达大学神经科学系）；DeepMind（深度思维）；Data Science Institute, Columbia University（哥伦比亚大学数据科学研究所）；Dept of Molecular & Cell Biology, Univ（大学分子与细胞生物学系）
Benjamin Peters, James J. DiCarlo, Todd Gureckis, Ralf Haefner, Leyla Isik, Joshua Tenenbaum, Talia Konkle, Thomas Naselaris, Kimberly Stachenfeld, Zenna Tavare… 展开作者
Benjamin Peters, James J. DiCarlo, Todd Gureckis, Ralf Haefner, Leyla Isik, Joshua Tenenbaum, Talia Konkle, Thomas Naselaris, Kimberly Stachenfeld, Zenna Tavares, Doris Tsao, Ilker Yildirim, Nikolaus Kriegeskorte
Vision is widely understood as an inference problem. However, two contrasting conceptions of the inference process have each been influential in research on biological vision as well as the engineering of machine vision. The first emphasizes bottom-up signal flow, describing vision as a largely feedforward, discriminative inference process that filters and transforms the visual information to remove irrelevant variation and represent behaviorally relevant information in a format suitable for downstream functions of cognition and behavioral control. In this conception, vision is driven by the sensory data, and perception is direct because the processing proceeds from the data to the latent variables of interest. The notion of "inference" in this conception is that of the engineering literature on neural networks, where feedforward convolutional neural networks processing images are said to perform inference. The alternative conception is that of vision as an inference process in Helmholtz's sense, where the sensory evidence is evaluated in the context of a generative model of the causal processes giving rise to it. In this conception, vision inverts a generative model through an interrogation of the evidence in a process often thought to involve top-down predictions of sensory data to evaluate the likelihood of alternative hypotheses. The authors include scientists rooted in roughly equal numbers in each of the conceptions and motivated to overcome what might be a false dichotomy between them and engage the other perspective in the realm of theory and experiment. The primate brain employs an unknown algorithm that may combine the advantages of both conceptions. We explain and clarify the terminology, review the key empirical evidence, and propose an empirical research program that transcends the dichotomy and sets the stage for revealing the mysterious hybrid algorithm of primate vision.
Brain stimulation is a core method in neuroscience. Numerous non-invasive brain stimulation (NIBS) techniques are currently in use in basic and clinical research, and recent advances promise the ability to non-invasively access deep brain structures. While encouraging, there is a surprising gap in our understanding of precisely how NIBS perturbs neural activity throughout an interconnected network, and how such perturbed neural activity ultimately links to behaviour. In this review, we will consider why non-human primate (NHP) models of NIBS are ideally situated to address this gap in knowledg…
Anatomical connectivity imposes strong constraints on brain function, but there is no general agreement about principles that govern its organization. Based on extensive quantitative data we tested the power of three models to predict connections of the primate cerebral cortex: architectonic similarity (structural model), spatial proximity (distance model) and thickness similarity (thickness model). Architectonic similarity showed the strongest and most consistent influence on connection features. This parameter was strongly associated with the presence or absence of inter-areal connections an…
Harvard Medical School（哈佛医学院）；National University of Singapore（新加坡国立大学）；Harvard University（哈佛大学）；Boston Children’s Hospital（波士顿儿童医院）；Center for Brains, Minds and Machines（大脑、心智与机器中心）
Artificial neural networks (ANNs) are considered the current best models of biological vision. ANNs are the best predictors of neural activity in the ventral stream; moreover, recent work has demonstrated that ANN models fitted to neuronal activity can guide the synthesis of images that drive pre-specified response patterns in small neuronal populations. Despite the success in predicting and steering firing activity, these results have not been connected with perceptual or behavioral changes. Here we propose an array of methods for creating minimal, targeted image perturbations that lead to ch…
Multi-modal neuroimaging projects are advancing our understanding of human brain architecture, function, connectivity using high-quality non-invasive data from many subjects. However, ground truth validation of connectivity using invasive tracers is not feasible in humans. Our NonHuman Primate Neuroimaging & Neuroanatomy Project (NHP_NNP) is an international effort (6 laboratories in 5 countries) to: (i) acquire and analyze high-quality multi-modal brain imaging data of macaque and marmoset monkeys using protocols and methods adapted from the HCP; (ii) acquire quantitative invasive tract-traci…
The hominid-specific non-LTR retrotransposon termed SINE VNTR Alu (SVA) is the youngest of the transposable elements in the human genome. The propagation of the most ancient SVA type A took place about thirteen millions years ago ago, and the youngest SVA types appeared in the human genome after the chimpanzee divergence. Functional enrichment analysis of genes associated with SVA insertions demonstrated their strong link to multiple ontological categories attributed to brain function and the disorders. SVA types that expanded their presence in the human genome at different stages of hominoid …
Huazhong University of Science and Technology（华中科技大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）；Stanford University（斯坦福大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Multi-view learning improves the learning performance by utilizing multi-view data: data collected from multiple sources, or feature sets extracted from the same data source. This approach is suitable for primate brain state decoding using cortical neural signals. This is because the complementary components of simultaneously recorded neural signals, local field potentials (LFPs) and action potentials (spikes), can be treated as two views. In this paper, we extended broad learning system (BLS), a recently proposed wide neural network architecture, from single-view learning to multi-view learni…
McGovern Institute for Brain Research, MIT（麻省理工学院麦戈文脑研究所）；KU Leuven（鲁汶大学）；Bay Labs Inc.（Bay Labs公司）；New York University（纽约大学）；Columbia University（哥伦比亚大学）；Stanford University（斯坦福大学）
Deep convolutional artificial neural networks (ANNs) are the leading class of candidate models of the mechanisms of visual processing in the primate ventral stream. While initially inspired by brain anatomy, over the past years, these ANNs have evolved from a simple eight-layer architecture in AlexNet to extremely deep and branching architectures, demonstrating increasingly better object categorization performance, yet bringing into question how brain-like they still are. In particular, typical deep models from the machine learning community are often hard to map onto the brain's anatomy due t…
The cognitive functions of human and non-human primates rely on the dynamic interplay of distributed neural assemblies. As such, it seems unlikely that cognition can be supported by macroscopic brain dynamics at the proximity of thermodynamic equilibrium. We confirmed this hypothesis by investigating electrocorticography data from non human primates undergoing different states of unconsciousness (sleep, and anesthesia with propofol, ketamine, and ketamine plus medetomidine), and funcional magnetic resonance imaging data from humans, both during deep sleep and under propofol anesthesia. Systema…
Recent methodological advances in MRI have enabled substantial growth in neuroimaging studies of non-human primates (NHPs), while open data-sharing through the PRIME-DE initiative has increased the availability of NHP MRI data and the need for robust multi-subject multi-center analyses. Streamlined acquisition and analysis protocols would accelerate and improve these efforts. However, consensus on minimal standards for data acquisition protocols and analysis pipelines for NHP imaging remains to be established, particularly for multi-center studies. Here, we draw parallels between NHP and human…
Neural correlations during a cognitive task are central to study brain information processing and computation. However, they have been poorly analyzed due to the difficulty of recording simultaneous single neurons during task performance. In the present work, we quantified neural directional correlations using spike trains that were simultaneously recorded in sensory, premotor, and motor cortical areas of two monkeys during a somatosensory discrimination task. Upon modeling spike trains as binary time series, we used a nonparametric Bayesian method to estimate pairwise directional correlations…
University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）；Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS)（三校神经影像与数据科学转化研究中心（TReNDS））；Georgia State University（佐治亚州立大学）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）
Independent component analysis (ICA) of multi-subject functional magnetic resonance imaging (fMRI) data has proven useful in providing a fully multivariate summary that can be used for multiple purposes. ICA can identify patterns that can discriminate between healthy controls (HC) and patients with various mental disorders such as schizophrenia (Sz). Temporal functional network connectivity (tFNC) obtained from ICA can effectively explain the interactions between brain networks. On the other hand, dictionary learning (DL) enables the discovery of hidden information in data using learnable basi…
Instituto Superior Tecnico（高等技术学院）；Universitat Politecnica de Catalunya（加泰罗尼亚理工大学）
The increasing access to brain signal data using electroencephalography creates new opportunities to study electrophysiological brain activity and perform ambulatory diagnoses of neuronal diseases. This work proposes a pairwise distance learning approach for Schizophrenia classification relying on the spectral properties of the signal. Given the limited number of observations (i.e. the case and/or control individuals) in clinical trials, we propose a Siamese neural network architecture to learn a discriminative feature space from pairwise combinations of observations per channel. In this way, …
Brain neural networks characterize various information propagation patterns for different emotional states. However, the statistical features based on traditional graph theory may ignore the spacial network difference. To reveal these inherent spatial features and increase the stability of emotional recognition, we proposed a hierarchical framework that can perform the multiple emotion recognitions with the multiple emotion-related spatial network topology patterns (MESNP) by combining a supervised learning with ensemble co-decision strategy. To evaluate the performance of our proposed MESNP a…
A major hurdle to clinical translation of brain-machine interfaces (BMIs) is that current decoders, which are trained from a small quantity of recent data, become ineffective when neural recording conditions subsequently change. We tested whether a decoder could be made more robust to future neural variability by training it to handle a variety of recording conditions sampled from months of previously collected data as well as synthetic training data perturbations. We developed a new multiplicative recurrent neural network BMI decoder that successfully learned a large variety of neural-to- kin…
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions（深港脑科学研究院-深圳基础研究机构）；Shenzhen Technological Research Center for Primate Translational Medicine（深圳灵长类转化医学技术研究中心）；University of Chinese Academy of Sciences（中国科学院大学）
Reading emotions precisely from segments of neural activity is crucial for the development of emotional brain-computer interfaces. Among all neural decoding algorithms, deep learning (DL) holds the potential to become the most promising one, yet progress has been limited in recent years. One possible reason is that the efficacy of DL strongly relies on training samples, yet the neural data used for training are often from non-human primates and mixed with plenty of noise, which in turn mislead the training of DL models. Given it is difficult to accurately determine animals' emotions from human…
Deep learning algorithms for predicting neuroimaging data have shown considerable promise in various applications. Prior work has demonstrated that deep learning models that take advantage of the data's 3D structure can outperform standard machine learning on several learning tasks. However, most prior research in this area has focused on neuroimaging data from adults. Within the Adolescent Brain and Cognitive Development (ABCD) dataset, a large longitudinal development study, we examine structural MRI data to predict gender and identify gender-related changes in brain structure. Results demon…
McGill University（麦吉尔大学）；Université de Montréal（蒙特利尔大学）
Neuroimaging data analysis often involves \emph{a-priori} selection of data features to study the underlying neural activity. Since this could lead to sub-optimal feature selection and thereby prevent the detection of subtle patterns in neural activity, data-driven methods have recently gained popularity for optimizing neuroimaging data analysis pipelines and thereby, improving our understanding of neural mechanisms. In this context, we developed a deep convolutional architecture that can identify discriminating patterns in neuroimaging data and applied it to electroencephalography (EEG) recor…
The social brain hypothesis postulates the increasing complexity of social interactions as a driving force for the evolution of cognitive abilities. Whereas dyadic and triadic relations play a basic role in defining social behaviours and pose many challenges for the social brain, individuals in animal societies typically belong to relatively large networks. How the structure and dynamics of these networks also contribute to the evolution of cognition, and vice versa, is less understood. Here we review how collective phenomena can occur in systems where social agents do not require sophisticate…
It has become increasingly popular to study the brain as a network due to the realization that functionality cannot be explained exclusively by independent activation of specialized regions. Instead, across a large spectrum of behaviors, function arises due to the dynamic interactions between brain regions. The existing literature on functional brain networks focuses mainly on a battery of network properties characterizing the "resting state" using for example the modularity, clustering, or path length among regions. In contrast, we seek to uncover subgraphs of functional connectivity that pre…
Istanbul Technical University（伊斯坦布尔技术大学）；Faculty of Medicine of Sousse（苏塞医学院）；Imperial College London（伦敦帝国学院）
The challenges of collecting medical data on neurological disorder diagnosis problems paved the way for learning methods with scarce number of samples. Due to this reason, one-shot learning still remains one of the most challenging and trending concepts of deep learning as it proposes to simulate the human-like learning approach in classification problems. Previous studies have focused on generating more accurate fingerprints of the population using graph neural networks (GNNs) with connectomic brain graph data. Thereby, generated population fingerprints named connectional brain template (CBTs…
Yale University（耶鲁大学）；Yale School of Medicine（耶鲁医学院）
Recurrent neural networks (RNNs) were designed for dealing with time-series data and have recently been used for creating predictive models from functional magnetic resonance imaging (fMRI) data. However, gathering large fMRI datasets for learning is a difficult task. Furthermore, network interpretability is unclear. To address these issues, we utilize multitask learning and design a novel RNN-based model that learns to discriminate between classes while simultaneously learning to generate the fMRI time-series data. Employing the long short-term memory (LSTM) structure, we develop a discrimina…
As a technology to read brain states from measurable brain activities, brain decoding are widely applied in industries and medical sciences. In spite of high demands in these applications for a universal decoder that can be applied to all individuals simultaneously, large variation in brain activities across individuals has limited the scope of many studies to the development of individual-specific decoders. In this study, we used deep neural network (DNN), a nonlinear hierarchical model, to construct a subject-transfer decoder. Our decoder is the first successful DNN-based subject-transfer de…
We present a method to discover interesting brain regions in [18F] fluorodeoxyglucose positron emission tomography (PET) scans, showing also the benefits when PET scans are in combined use with non-imaging variables. The discriminative brain regions facilitate a better understanding of Alzheimer's disease (AD) progression, and they can also be used for predicting conversion from mild cognitive impairment (MCI) to AD. A survival analysis(Cox regression) and infinite Gaussian mixture model (IGMM) are introduced to identify the informative brain regions, which can be further used to make a predic…
