The unreasonable effectiveness of small neural ensembles in high-dimensional brain
University of Leicester（莱斯特大学）；Lobachevsky University（罗巴切夫斯基大学）；Universidad Complutense de Madrid（马德里康普顿斯大学）；Saint-Petersburg State Electrotechnical University（圣彼得堡国立电工大学）
A. N. Gorban, V. A. Makarov, I. Y. Tyukin
备注：
Review paper, accepted in Physics of Life Reviews; minor corrections
发表信息：
Physics of Life Reviews Volume 29, July 2019, Pages 55-88
Despite the widely-spread consensus on the brain complexity, sprouts of the single neuron revolution emerged in neuroscience in the 1970s. They brought many unexpected discoveries, including grandmother or concept cells and sparse coding of information in the brain.
In machine learning for a long time, the famous curse of dimensionality seemed to be an unsolvable problem. Nevertheless, the idea of the blessing of dimensionality becomes gradually more and more popular. Ensembles of non-interacting or weakly interacting simple units prove to be an effective tool for solving essentially multidimensional problems. This approach is especially useful for one-shot (non-iterative) correction of errors in large legacy artificial intelligence systems.
These simplicity revolutions in the era of complexity have deep fundamental reasons grounded in geometry of multidimensional data spaces. To explore and understand these reasons we revisit the background ideas of statistical physics. In the course of the 20th century they were developed into the concentration of measure theory. New stochastic separation theorems reveal the fine structure of the data clouds.
We review and analyse biological, physical, and mathematical problems at the core of the fundamental question: how can high-dimensional brain organise reliable and fast learning in high-dimensional world of data by simple tools?
Two critical applications are reviewed to exemplify the approach: one-shot correction of errors in intellectual systems and emergence of static and associative memories in ensembles of single neurons.
Salk Institute（索尔克研究所）；University of California San Diego（加利福尼亚大学圣迭戈分校）
Deep learning networks have been trained to recognize speech, caption photographs and translate text between languages at high levels of performance. Although applications of deep learning networks to real world problems have become ubiquitous, our understanding of why they are so effective is lacking. These empirical results should not be possible according to sample complexity in statistics and non-convex optimization theory. However, paradoxes in the training and effectiveness of deep learning networks are being investigated and insights are being found in the geometry of high-dimensional s…
University of Leicester（莱斯特大学）；Lobachevsky University（罗巴切夫斯基大学）；Universidad Complutense de Madrid（马德里康普顿斯大学）；Saint-Petersburg State Electrotechnical University（圣彼得堡国立电工大学）
This paper is the final part of the scientific discussion organised by the Journal "Physics of Life Rviews" about the simplicity revolution in neuroscience and AI. This discussion was initiated by the review paper "The unreasonable effectiveness of small neural ensembles in high-dimensional brain". Phys Life Rev 2019, doi 10.1016/j.plrev.2018.09.005, arXiv:1809.07656. The topics of the discussion varied from the necessity to take into account the difference between the theoretical random distributions and "extremely non-random" real distributions and revise the common machine learning theory, …
Ensemble learning is a classical learning method utilizing a group of weak learners to form a strong learner, which aims to increase the accuracy of the model. Recently, brain-inspired hyperdimensional computing (HDC) becomes an emerging computational paradigm that has achieved success in various domains such as human activity recognition, voice recognition, and bio-medical signal classification. HDC mimics the brain cognition and leverages high-dimensional vectors (e.g., 10000 dimensions) with fully distributed holographic representation and (pseudo-)randomness. This paper presents the first …
School of Computing and Engineering, University of West London（西伦敦大学计算与工程学院）
Objectives: Functional connectivity triggered by naturalistic stimulus (e.g., movies) and machine learning techniques provide a great insight in exploring the brain functions such as fluid intelligence. However, functional connectivity are considered to be multi-layered, while traditional machine learning based on individual models not only are limited in performance, but also fail to extract multi-dimensional and multi-layered information from brain network. Methods: In this study, inspired by multi-layer brain network structure, we propose a new method namely Weighted Ensemble-model and Netw…
Zhejiang University School of Medicine（浙江大学医学院）；Affiliated Mental Health Center & Hangzhou Seventh People’s Hospital（附属精神卫生中心暨杭州市第七人民医院）；Qiushi Academy for Advanced Studies, Zhejiang University（浙江大学求是高等研究院）；Zhejiang Lab（之江实验室）；Second Affiliated Hospital of Zhejiang University School of Medicine（浙江大学医学院附属第二医院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Objective: Brain-machine interfaces (BMIs) aim to provide direct brain control of devices such as prostheses and computer cursors, which have demonstrated great potential for mobility restoration. One major limitation of current BMIs lies in the unstable performance in online control due to the variability of neural signals, which seriously hinders the clinical availability of BMIs. Method: To deal with the neural variability in online BMI control, we propose a dynamic ensemble Bayesian filter (DyEnsemble). DyEnsemble extends Bayesian filters with a dynamic measurement model, which adjusts its…
Recently, practical brain-computer interface is actively carried out, especially, in an ambulatory environment. However, the electroencephalography (EEG) signals are distorted by movement artifacts and electromyography signals when users are moving, which make hard to recognize human intention. In addition, as hardware issues are also challenging, ear-EEG has been developed for practical brain-computer interface and has been widely used. In this paper, we proposed ensemble-based convolutional neural networks in ambulatory environment and analyzed the visual event-related potential responses in…
Biomedical Image Analysis Group, Imperial College London（伦敦帝国理工学院生物医学图像分析组）
Deep learning approaches such as convolutional neural nets have consistently outperformed previous methods on challenging tasks such as dense, semantic segmentation. However, the various proposed networks perform differently, with behaviour largely influenced by architectural choices and training settings. This paper explores Ensembles of Multiple Models and Architectures (EMMA) for robust performance through aggregation of predictions from a wide range of methods. The approach reduces the influence of the meta-parameters of individual models and the risk of overfitting the configuration to a …
The growing importance and utilization of measuring brain waves (e.g. EEG signals of eye state) in brain-computer interface (BCI) applications highlighted the need for suitable classification methods. In this paper, a comparison between three of well-known classification methods (i.e. support vector machine (SVM), hidden Markov map (HMM), and radial basis function (RBF)) for EEG based eye state classification was achieved. Furthermore, a suggested method that is based on ensemble model was tested. The suggested (ensemble system) method based on a voting algorithm with two kernels: random fores…
FlyPrompt: Brain-Inspired Random-Expanded Routing with Temporal-Ensemble Experts for General Continual Learning
School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University（生命科学学院，IDG/麦克格雷文脑科学研究院，清华大学）；Department of Psychological and Cognitive Sciences, Tsinghua University（心理学与认知科学系，清华大学）；School of Medicine, Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区医学学院）
Texas Tech Neuroimaging Institute, Texas Tech University（德克萨斯理工大学神经影像研究所，德克萨斯理工大学）
Accurate diagnosis of Alzheimer's disease (AD) is both challenging and time consuming. With a systematic approach for early detection and diagnosis of AD, steps can be taken towards the treatment and prevention of the disease. This study explores the practical application of deep learning models for diagnosis of AD. Due to computational complexity, large training times and limited availability of labelled dataset, a 3D full brain CNN (convolutional neural network) is not commonly used, and researchers often prefer 2D CNN variants. In this study, full brain 3D version of well-known 2D CNNs were…
This paper continues the research that considers a new cognitive model based strongly on the human brain. In particular, it considers the neural binding structure of an earlier paper. It also describes some new methods in the areas of image processing and behaviour simulation. The work is all based on earlier research by the author and the new additions are intended to fit in with the overall design. For image processing, a grid-like structure is used with 'full linking'. Each cell in the classifier grid stores a list of all other cells it gets associated with and this is used as the learned i…
Convolutional neural networks (CNNs) are increasingly being used to automate segmentation of organs-at-risk in radiotherapy. Since large sets of highly curated data are scarce, we investigated how much data is required to train accurate and robust head and neck auto-segmentation models. For this, an established 3D CNN was trained from scratch with different sized datasets (25-1000 scans) to segment the brainstem, parotid glands and spinal cord in CTs. Additionally, we evaluated multiple ensemble techniques to improve the performance of these models. The segmentations improved with training set…
Enhancing Early Alzheimer Disease Detection through Big Data and Ensemble Few-Shot Learning
Robotics and Internet of Things Laboratory, Prince Sultan University（王子苏大学机器人与物联网实验室）；RIADI Laboratory, National School of Computer Science, University of Manouba（RIADI实验室，国家计算机科学学院，曼努巴大学）
