The non-uniform sampling is a powerful approach to enable fast acquisition but requires sophisticated reconstruction algorithms. Faithful reconstruction from partial sampled exponentials is highly expected in general signal processing and many applications. Deep learning has shown astonishing potential in this field but many existing problems, such as lack of robustness and explainability, greatly limit its applications. In this work, by combining merits of the sparse model-based optimization method and data-driven deep learning, we propose a deep learning architecture for spectra reconstruction from undersampled data, called MoDern. It follows the iterative reconstruction in solving a sparse model to build the neural network and we elaborately design a learnable soft-thresholding to adaptively eliminate the spectrum artifacts introduced by undersampling. Extensive results on both synthetic and biological data show that MoDern enables more robust, high-fidelity, and ultra-fast reconstruction than the state-of-the-art methods. Remarkably, MoDern has a small number of network parameters and is trained on solely synthetic data while generalizing well to biological data in various scenarios. Furthermore, we extend it to an open-access and easy-to-use cloud computing platform (XCloud-MoDern), contributing a promising strategy for further development of biological applications.
We introduce bio-inspired artificial neural networks consisting of neurons that are additionally characterized by spatial positions. To simulate properties of biological systems we add the costs penalizing long connections and the proximity of neurons in a two-dimensional space. Our experiments show that in the case where the network performs two different tasks, the neurons naturally split into clusters, where each cluster is responsible for processing a different task. This behavior not only corresponds to the biological systems, but also allows for further insight into interpretability or c…
Canadian Centre for Behavioural Neuroscience, University of Lethbridge（莱斯布里奇大学加拿大行为神经科学中心）
Since humans still outperform artificial neural networks on many tasks, drawing inspiration from the brain may help to improve current machine learning algorithms. Contrastive Hebbian Learning (CHL) and Equilibrium Propagation (EP) are biologically plausible algorithms that update weights using only local information (without explicitly calculating gradients) and still achieve performance comparable to conventional backpropagation. In this study, we augmented CHL and EP with Adjusted Adaptation, inspired by the adaptation effect observed in neurons, in which a neuron's response to a given stim…
University of Tübingen（图宾根大学）；Bernstein Center for Computational Neuroscience（伯恩斯坦计算神经科学中心）；Max Planck Institute for Biological Cybernetics（马克斯·普朗克生物控制论研究所）
A recent paper suggests that Deep Neural Networks can be protected from gradient-based adversarial perturbations by driving the network activations into a highly saturated regime. Here we analyse such saturated networks and show that the attacks fail due to numerical limitations in the gradient computations. A simple stabilisation of the gradient estimates enables successful and efficient attacks. Thus, it has yet to be shown that the robustness observed in highly saturated networks is not simply due to numerical limitations.
CiNet, National Institute of Information and Communications Technology（信息通信研究机构脑信息通信融合研究中心）；Graduate School of Information Science and Technology, Osaka University（大阪大学信息科学与技术研究科）
In this study, we propose a novel deep neural network and its supervised learning method that uses a feedforward supervisory signal. The method is inspired by the human visual system and performs human-like association-based learning without any backward error propagation. The feedforward supervisory signal that produces the correct result is preceded by the target signal and associates its confirmed label with the classification result of the target signal. It effectively uses a large amount of information from the feedforward signal, and forms a continuous and rich learning representation. T…
School of Mathematical Sciences（数学科学学院）；School of Psychology, University of Nottingham（诺丁汉大学心理学院）
When training neural networks for classification tasks with backpropagation, parameters are updated on every trial, even if the sample is classified correctly. In contrast, humans concentrate their learning effort on errors. Inspired by human learning, we introduce lazy learning, which only learns on incorrect samples. Lazy learning can be implemented in a few lines of code and requires no hyperparameter tuning. Lazy learning achieves state-of-the-art performance and is particularly suited when datasets are large. For instance, it reaches 99.2% test accuracy on Extended MNIST using a single-la…
Harvard University（哈佛大学）；Massachusetts Institute of Technology（麻省理工学院）
A convolutional neural network strongly robust to adversarial perturbations at reasonable computational and performance cost has not yet been demonstrated. The primate visual ventral stream seems to be robust to small perturbations in visual stimuli but the underlying mechanisms that give rise to this robust perception are not understood. In this work, we investigate the role of two biologically plausible mechanisms in adversarial robustness. We demonstrate that the non-uniform sampling performed by the primate retina and the presence of multiple receptive fields with a range of receptive fiel…
There has been a strong push recently to examine biological scale simulations of neuromorphic algorithms to achieve stronger inference capabilities. This paper presents a set of piecewise linear spiking neuron models, which can reproduce different behaviors, similar to the biological neuron, both for a single neuron as well as a network of neurons. The proposed models are investigated, in terms of digital implementation feasibility and costs, targeting large scale hardware implementation. Hardware synthesis and physical implementations on FPGA show that the proposed models can produce precise …
Singapore University of Technology and Design（新加坡科技设计大学）；Tsinghua University（清华大学）
Biological systems in nature have evolved for millions of years to adapt and survive the environment. Many features they developed can be inspirational and beneficial for solving technical problems in modern industries. This leads to a novel form of design-by-analogy called bio-inspired design (BID). Although BID as a design method has been proven beneficial, the gap between biology and engineering continuously hinders designers from effectively applying the method. Therefore, we explore the recent advance of artificial intelligence (AI) for a computational approach to bridge the gap. This pap…
Inspired by biophysical principles underlying nonlinear dendritic computation in neural circuits, we develop a scheme to train deep neural networks to make them robust to adversarial attacks. Our scheme generates highly nonlinear, saturated neural networks that achieve state of the art performance on gradient based adversarial examples on MNIST, despite never being exposed to adversarially chosen examples during training. Moreover, these networks exhibit unprecedented robustness to targeted, iterative schemes for generating adversarial examples, including second-order methods. We further ident…
Oja's rule [Oja, Journal of mathematical biology 1982] is a well-known biologically-plausible algorithm using a Hebbian-type synaptic update rule to solve streaming principal component analysis (PCA). Computational neuroscientists have known that this biological version of Oja's rule converges to the top eigenvector of the covariance matrix of the input in the limit. However, prior to this work, it was open to prove any convergence rate guarantee. In this work, we give the first convergence rate analysis for the biological version of Oja's rule in solving streaming PCA. Moreover, our convergen…
Mila - Quebec Artificial Intelligence Institute（米拉——魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）
Dynamic adaptation in single-neuron response plays a fundamental role in neural coding in biological neural networks. Yet, most neural activation functions used in artificial networks are fixed and mostly considered as an inconsequential architecture choice. In this paper, we investigate nonlinear activation function adaptation over the large time scale of learning, and outline its impact on sequential processing in recurrent neural networks. We introduce a novel parametric family of nonlinear activation functions, inspired by input-frequency response curves of biological neurons, which allows…
Shenzhen University（深圳大学）；Army Engineering University of People’s Liberation Army of China（中国人民解放军陆军工程大学）；Hangzhou Dianzi University（杭州电子科技大学）
Most of the Zero-Shot Learning (ZSL) algorithms currently use pre-trained models as their feature extractors, which are usually trained on the ImageNet data set by using deep neural networks. The richness of the feature information embedded in the pre-trained models can help the ZSL model extract more useful features from its limited training samples. However, sometimes the difference between the training data set of the current ZSL task and the ImageNet data set is too large, which may lead to the use of pre-trained models has no obvious help or even negative impact on the performance of the …
Carnegie Mellon University（卡内基梅隆大学）；Stanford University（斯坦福大学）；UnifyID AI Labs（UnifyID人工智能实验室）
The interplay between inter-neuronal network topology and cognition has been studied deeply by connectomics researchers and network scientists, which is crucial towards understanding the remarkable efficacy of biological neural networks. Curiously, the deep learning revolution that revived neural networks has not paid much attention to topological aspects. The architectures of deep neural networks (DNNs) do not resemble their biological counterparts in the topological sense. We bridge this gap by presenting initial results of Deep Connectomics Networks (DCNs) as DNNs with topologies inspired b…
Axelera AI；IBM Research, Zurich（IBM苏黎世研究院）；Graz University of Technology（格拉茨工业大学）；ETH Zurich（苏黎世联邦理工学院）；IBM Research AI, Yorktown Heights（IBM研究院）
Automatic speech recognition (ASR) is a capability which enables a program to process human speech into a written form. Recent developments in artificial intelligence (AI) have led to high-accuracy ASR systems based on deep neural networks, such as the recurrent neural network transducer (RNN-T). However, the core components and the performed operations of these approaches depart from the powerful biological counterpart, i.e., the human brain. On the other hand, the current developments in biologically-inspired ASR models, based on spiking neural networks (SNNs), lag behind in terms of accurac…
Huawei Technologies（华为技术有限公司）；Southeast University（东南大学）；Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education（计算机网络和信息集成教育部重点实验室（东南大学））；Collaborative Innovation Center of Wireless Communications Technology（无线通信技术协同创新中心）
Numerous researches have proved that deep neural networks (DNNs) can fit everything in the end even given data with noisy labels, and result in poor generalization performance. However, recent studies suggest that DNNs tend to gradually memorize the data, moving from correct data to mislabeled data. Inspired by this finding, we propose a novel method named Dynamic Loss Thresholding (DLT). During the training process, DLT records the loss value of each sample and calculates dynamic loss thresholds. Specifically, DLT compares the loss value of each sample with the current loss threshold. Samples…
Energy-Efficient Information Representation in MNIST Classification Using Biologically Inspired Learning
KEIM Institute, Albstadt-Sigmaringen University, Germany（凯姆研究所，阿尔布施塔特-西格马林根大学，德国）；Department of Computer Science, Chemnitz University of Technology, Germany（计算机科学系，化学工业大学，德国）
SPAWAR Systems Center Pacific（美国海军空间与海战系统司令部太平洋系统中心）；Georgia Institute of Technology（佐治亚理工学院）；ID Analytics（ID分析公司）
Automatic modulation classification (AMC) is an important task for modern communication systems; however, it is a challenging problem when signal features and precise models for generating each modulation may be unknown. We present a new biologically-inspired AMC method without the need for models or manually specified features --- thus removing the requirement for expert prior knowledge. We accomplish this task using regularized stacked sparse denoising autoencoders (SSDAs). Our method selects efficient classification features directly from raw in-phase/quadrature (I/Q) radio signals in an un…
Deep reinforcement learning requires a heavy price in terms of sample efficiency and overparameterization in the neural networks used for function approximation. In this work, we use tensor factorization in order to learn more compact representation for reinforcement learning policies. We show empirically that in the low-data regime, it is possible to learn online policies with 2 to 10 times less total coefficients, with little to no loss of performance. We also leverage progress in second order optimization, and use the theory of wavelet scattering to further reduce the number of learned coef…
Institute for Systems Research, University of Maryland（系统研究所以及马里兰大学）；Google Inc（谷歌公司）
Recent advancements in deep learning have led to drastic improvements in speech segregation models. Despite their success and growing applicability, few efforts have been made to analyze the underlying principles that these networks learn to perform segregation. Here we analyze the role of harmonicity on two state-of-the-art Deep Neural Networks (DNN)-based models- Conv-TasNet and DPT-Net. We evaluate their performance with mixtures of natural speech versus slightly manipulated inharmonic speech, where harmonics are slightly frequency jittered. We find that performance deteriorates significant…
Evolving Afferent Architectures: Biologically-inspired Models for Damage-Avoidance Learning
Saarland University（萨尔兰大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Statistics and Machine Learning, German Center for Neurodegenerative Diseases (DZNE)（神经退行性疾病德国研究中心统计与机器学习部门）；MRC Biostatistics Unit, University of Cambridge（剑桥大学医学研究委员会生物统计学单位）
Invariances to translation, rotation and other spatial transformations are a hallmark of the laws of motion, and have widespread use in the natural sciences to reduce the dimensionality of systems of equations. In supervised learning, such as in image classification tasks, rotation, translation and scale invariances are used to augment training datasets. In this work, we use data augmentation in a similar way, exploiting symmetry in the quadruped domain of the DeepMind control suite (Tassa et al. 2018) to add to the trajectories experienced by the actor in the actor-critic algorithm of Abdolma…
Institute of Automation, Chinese Academy of Sciences (CAS)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Spiking Neural Networks (SNNs) can do inference with low power consumption due to their spike sparsity. ANN-SNN conversion is an efficient way to achieve deep SNNs by converting well-trained Artificial Neural Networks (ANNs). However, the existing methods commonly use constant threshold for conversion, which prevents neurons from rapidly delivering spikes to deeper layers and causes high time delay. In addition, the same response for different inputs may result in information loss during the information transmission. Inspired by the biological model mechanism, we propose a multi-stage adaptive…
More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routing
University of Colorado Boulder（科罗拉多大学博尔德分校）；National Institute of Mental Health（美国国家精神卫生研究所）；Johannes Gutenberg University Mainz（美因茨约翰内斯·古腾堡大学）；University of Chicago（芝加哥大学）；Google DeepMind（谷歌DeepMind）
NeuroPlastic: A Plasticity-Modulated Optimizer for Biologically Inspired Learning Dynamics
Department of Neurology, Beth Israel Deaconess Medical Center, Harvard Medical School（神经病学系，贝塞斯达德acons医疗中心，哈佛医学院）；Department of Biostatistics, Harvard T.H. Chan School of Public Health（生物统计学系，哈佛T.H. Chan公共卫生学院）；Department of Statistics, John A. Paulson School of Engineering and Applied Sciences, Harvard University（统计学系，约翰·A·保罗森工程与应用科学学院，哈佛大学）；Department of Psychology, Columbia University（心理学系，哥伦比亚大学）
The ability to look multiple times through a series of pose-adjusted glimpses is fundamental to human vision. This critical faculty allows us to understand highly complex visual scenes. Short term memory plays an integral role in aggregating the information obtained from these glimpses and informing our interpretation of the scene. Computational models have attempted to address glimpsing and visual attention but have failed to incorporate the notion of memory. We introduce a novel, biologically inspired visual working memory architecture that we term the Hebb-Rosenblatt memory. We subsequently…
Recurrent neural networks (RNNs) provide state-of-the-art performances in a wide variety of tasks that require memory. These performances can often be achieved thanks to gated recurrent cells such as gated recurrent units (GRU) and long short-term memory (LSTM). Standard gated cells share a layer internal state to store information at the network level, and long term memory is shaped by network-wide recurrent connection weights. Biological neurons on the other hand are capable of holding information at the cellular level for an arbitrary long amount of time through a process called bistability…
