We experimentally demonstrate classification of 4x4 binary images into 4 classes, using a 3-layer mixed-signal neuromorphic network ("MLP perceptron"), based on two passive 20x20 memristive crossbar arrays, board-integrated with discrete CMOS components. The network features 10 hidden-layer and 4 output-layer analog CMOS neurons and 428 metal-oxide memristors, i.e. is almost an order of magnitude more complex than any previously reported functional memristor circuit. Moreover, the inference operation of this classifier is performed entirely in the integrated hardware. To deal with larger crossbar arrays, we have developed a semi-automatic approach to their forming and testing, and compared several memristor training schemes for coping with imperfect behavior of these devices, as well as with variability of analog CMOS neurons. The effectiveness of the proposed schemes for defect and variation tolerance was verified experimentally using the implemented network and, additionally, by modeling the operation of a larger network, with 300 hidden-layer neurons, on the MNIST benchmark. Finally, we propose a simple modification of the implemented memristor-based vector-by-matrix multiplier to allow its operation in a wider temperature range.
In this position paper, we present a discussion on neuromorphic computing and especially the learning/training algorithm to design a series of brains with different memristive values to solve complex ill-posed inverse problems based on a Finite Element(FE) method. First, the neuromorphic computing is addressed and we focus on a type of memristive circuit computing that falls into the scope of neuromorphic computing. Secondly based on reference [1] in which the complex dynamics of the complex memristive circuit was studied, we design a method and an approach to train the memristive circuit so t…
Implementation of Neuromorphic Systems using post Complementary Metal-Oxide-Semiconductor (CMOS) technology based Memristive Crossbar Array (MCA) has emerged as a promising solution to enable low-power acceleration of neural networks. However, the recent trend to design Deep Neural Networks (DNNs) for achieving human-like cognitive abilities poses significant challenges towards the scalable design of neuromorphic systems (due to the increase in computation/storage demands). Network pruning [7] is a powerful technique to remove redundant connections for designing optimally connected (maximally …
In this paper, we propose an efficient predefined structured sparsity-based ex-situ training framework for a hybrid CMOS-memristive neuromorphic hardware for deep neural network to significantly lower the power consumption and computational complexity and improve scalability. The structure is verified on a wide range of datasets including MNIST handwritten recognition, breast cancer prediction, and mobile health monitoring. The results of this study show that compared to its fully connected version, the proposed structure provides significant power reduction while maintaining high classificati…
Recent breakthroughs in neuromorphic computing show that local forms of gradient descent learning are compatible with Spiking Neural Networks (SNNs) and synaptic plasticity. Although SNNs can be scalably implemented using neuromorphic VLSI, an architecture that can learn using gradient-descent in situ is still missing. In this paper, we propose a local, gradient-based, error-triggered learning algorithm with online ternary weight updates. The proposed algorithm enables online training of multi-layer SNNs with memristive neuromorphic hardware showing a small loss in the performance compared wit…
Neuromorphic computing has emerged as a promising avenue towards building the next generation of intelligent computing systems. It has been proposed that memristive devices, which exhibit history-dependent conductivity modulation, could efficiently represent the synaptic weights in artificial neural networks. However, precise modulation of the device conductance over a wide dynamic range, necessary to maintain high network accuracy, is proving to be challenging. To address this, we present a multi-memristive synaptic architecture with an efficient global counter-based arbitration scheme. We fo…
Bielefeld University（比勒费尔德大学）；University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）
The development of memristive device technologies has reached a level of maturity to enable the design of complex and large-scale hybrid memristive-CMOS neural processing systems. These systems offer promising solutions for implementing novel in-memory computing architectures for machine learning and data analysis problems. We argue that they are also ideal building blocks for the integration in neuromorphic electronic circuits suitable for ultra-low power brain-inspired sensory processing systems, therefore leading to the innovative solutions for always-on edge-computing and Internet-of-Thing…
Neuromorphic computing based on spiking neural networks has the potential to significantly improve on-line learning capabilities and energy efficiency of artificial intelligence, specially for edge computing. Recent progress in computational neuroscience have demonstrated the importance of heterosynaptic plasticity for network activity regulation and memorization. Implementing heterosynaptic plasticity in hardware is thus highly desirable, but important materials and engineering challenges remain, calling for breakthroughs in neuromorphic devices. In this mini-review, we propose an overview of…
Memristive devices represent a promising technology for building neuromorphic electronic systems. In addition to their compactness and non-volatility features, they are characterized by computationally relevant physical properties, such as state-dependence, non-linear conductance changes, and intrinsic variability in both their switching threshold and conductance values, that make them ideal devices for emulating the bio-physics of real synapses. In this paper we present a spiking neural network architecture that supports the use of memristive devices as synaptic elements, and propose mixed-si…
We present MEMprop, the adoption of gradient-based learning to train fully memristive spiking neural networks (MSNNs). Our approach harnesses intrinsic device dynamics to trigger naturally arising voltage spikes. These spikes emitted by memristive dynamics are analog in nature, and thus fully differentiable, which eliminates the need for surrogate gradient methods that are prevalent in the spiking neural network (SNN) literature. Memristive neural networks typically either integrate memristors as synapses that map offline-trained networks, or otherwise rely on associative learning mechanisms t…
The areal footprint of memristors is a key consideration in material-based neuromorophic computing and large-scale architecture integration. Electronic transport in the most widely investigated memristive devices is mediated by filaments, posing a challenge to their scalability in architecture implementation. Here we present a compelling alternative memristive device and demonstrate that areal downscaling leads to enhancement in memristive memory window, while maintaining analogue behavior, contrary to expectations. Our device designs directly integrated on semiconducting Nb-SrTiO$_3$ allows l…
In this paper, the hardware implementation of a neuromorphic system is presented. This system is composed of a Leaky Integrate-and-Fire with Latency (LIFL) neuron and a Spike-Timing Dependent Plasticity (STDP) synapse. LIFL neuron model allows to encode more information than the common Integrate-and-Fire models, typically considered for neuromorphic implementations. In our system LIFL neuron is implemented using CMOS circuits while memristor is used for the implementation of the STDP synapse. A description of the entire circuit is provided. Finally, the capabilities of the proposed architectur…
Nazarbayev University（纳扎尔巴耶夫大学）；University of California Berkeley（加州大学伯克利分校）
The volume, veracity, variability, and velocity of data produced from the ever-increasing network of sensors connected to Internet pose challenges for power management, scalability, and sustainability of cloud computing infrastructure. Increasing the data processing capability of edge computing devices at lower power requirements can reduce several overheads for cloud computing solutions. This paper provides the review of neuromorphic CMOS-memristive architectures that can be integrated into edge computing devices. We discuss why the neuromorphic architectures are useful for edge devices and s…
At the Faraday Discussion, in the paper titled `Neuromorphic computation with spiking memristors: habituation, experimental instantiation of logic gates and a novel sequence-sensitive perceptron model' it was demonstrated that a large amount of computation could be done in a sequential way using memristor current spikes (d.c. response). As these spikes are found in many memristors (possibly all), this novel approach could be highly useful for fast and reproducible memristor circuits. However, questions were raised as to whether these spikes were actually due to memristance or merely capacitanc…
The automated wafer inspection and quality control is a complex and time-consuming task, which can speed up using neuromorphic memristive architectures, as a separate inspection device or integrating directly into sensors. This paper presents the performance analysis and comparison of different neuromorphic architectures for patterned wafer quality inspection and classification. The application of non-volatile memristive devices in these architectures ensures low power consumption, small on-chip area scalability. We demonstrate that Long-Short Term Memory (LSTM) outperforms other architectures…
The advances in the field of machine learning using neuromorphic systems have paved the pathway for extensive research on possibilities of hardware implementations of neural networks. Various memristive technologies such as oxide-based devices, spintronics and phase change materials have been explored to implement the core functional units of neuromorphic systems, namely the synaptic network, and the neuronal functionality, in a fast and energy efficient manner. However, various non-idealities in the crossbar implementations of the synaptic arrays can significantly degrade performance of neura…
Memristive neural networks (MNNs), which use memristors as neurons or synapses, have become a hot research topic recently. However, most memristors are not compatible with mainstream integrated circuit technology and their stabilities in large-scale are not very well so far. In this paper, a hardware friendly MNN circuit is introduced, in which the memristive characteristics are implemented by digital integrated circuit. Through this method, spike timing dependent plasticity (STDP) and unsupervised learning are realized. A weight sharing mechanism is proposed to bridge the gap of network scale…
Neuromorphic systems that learn and predict from streaming inputs hold significant promise in pervasive edge computing and its applications. In this paper, a neuromorphic system that processes spatio-temporal information on the edge is proposed. Algorithmically, the system is based on hierarchical temporal memory that inherently offers online learning, resiliency, and fault tolerance. Architecturally, it is a full custom mixed-signal design with an underlying digital communication scheme and analog computational modules. Therefore, the proposed system features reconfigurability, real-time proc…
Synchronization of two memristive coupled van der Pol oscillators
本文的目的是探索通过由Ag-TiO x-Al忆阻器件组成的阻容网络耦合两个van der Pol（vdP）振荡器的可能性。通过网络连接两个可编程单结晶体管（PUT）的栅极端子来介导耦合。在高阻态（HRS）的memresistance是在MOhm的顺序，导致两个独立的自持振荡器，其特征在于不同的频率f1和f2和振荡之间没有相位关系。在几个周期之后并且依赖于介导的脉冲幅度，忆阻器件切换到低电阻状态（LRS），并且观察到频率适应和相位锁定。实验结果强调从理论上考虑两个耦合的vdP方程的系统。所提出的神经形态回路传达了两个相互作用的神经元集合的基本原则：同步和记忆。该实验可能通向更大的神经形态网络，其中耦合参数可以随时间和强度而变化，并且由忆阻器件实现。
The progress in the field of neural computation hinges on the use of hardware more efficient than the conventional microprocessors. Recent works have shown that mixed-signal integrated memristive circuits, especially their passive ('0T1R') variety, may increase the neuromorphic network performance dramatically, leaving far behind their digital counterparts. The major obstacle, however, is relatively immature memristor technology so that only limited functionality has been demonstrated to date. Here we experimentally demonstrate operation of one-hidden layer perceptron classifier entirely in th…
Spiking recurrent neural networks (RNNs) are a promising tool for solving a wide variety of complex cognitive and motor tasks, due to their rich temporal dynamics and sparse processing. However training spiking RNNs on dedicated neuromorphic hardware is still an open challenge. This is due mainly to the lack of local, hardware-friendly learning mechanisms that can solve the temporal credit assignment problem and ensure stable network dynamics, even when the weight resolution is limited. These challenges are further accentuated, if one resorts to using memristive devices for in-memory computing…
Brain-inspired computing proposes a set of algorithmic principles that hold promise for advancing artificial intelligence. They endow systems with self learning capabilities, efficient energy usage, and high storage capacity. A core concept that lies at the heart of brain computation is sequence learning and prediction. This form of computation is essential for almost all our daily tasks such as movement generation, perception, and language. Understanding how the brain performs such a computation is not only important to advance neuroscience but also to pave the way to new technological brain-…
The emergence of nano-scale memristive devices encouraged many different research areas to exploit their use in multiple applications. One of the proposed applications was to implement synaptic connections in bio-inspired neuromorphic systems. Large-scale neuromorphic hardware platforms are being developed with increasing number of neurons and synapses, having a critical bottleneck in the online learning capabilities. Spike-timing-dependent plasticity (STDP) is a widely used learning mechanism inspired by biology which updates the synaptic weight as a function of the temporal correlation betwe…
Atomic Switch Networks (ASN) comprising silver iodide (AgI) junctions, a material previously unexplored as functional memristive elements within highly-interconnected nanowire networks, were employed as a neuromorphic substrate for physical Reservoir Computing (RC). This new class of ASN-based devices has been physically characterized and utilized to classify spoken digit audio data, demonstrating the utility of substrate-based device architectures where intrinsic material properties can be exploited to perform computation in-materio. This work demonstrates high accuracy in the classification …
Memristive materials are related to neuromorphic applications as they can combine information processing with memory storage in a single computational element, just as biological neurons. Many of these bioinspired materials emulate the characteristics of memory and learning processes that happen in the brain. In this work, we report the memristive properties of a two-terminal (2-T) organic device based on ionic migration mediated by an ion-transport polymer. The material possesses unique memristive properties: it is reversibly switchable, shows tens of conductive states, presents Hebbian learn…
Neuromorphic computing using spike-based learning has broad prospects in reducing computing power. Memristive neurons composed with two locally active memristors have been used to mimic the dynamical behaviors of biological neurons. In this work, the dynamic operating conditions of NbO2-based memristive neurons and their transformation boundaries between the spiking and the bursting are comprehensively investigated. Furthermore, the underlying mechanism of bursting is analyzed and the controllability of the number of spikes during each burst period is demonstrated. Finally, pattern classificat…
Learning Chaotic Dynamics with Neuromorphic Network Dynamics
School of Physics, University of Sydney, Sydney, NSW 2006, Australia.（悉尼大学物理系）；School of Mathematics and Statistics, University of Sydney, Sydney, NSW 2006, Australia（悉尼大学数学与统计学系）；Centre for Complex Systems, University of Sydney, Sydney, NSW 2006, Australia.（悉尼大学复杂系统中心）
Hierarchical Temporal Memory (HTM) is a neuromorphic algorithm that emulates sparsity, hierarchy and modularity resembling the working principles of neocortex. Feature encoding is an important step to create sparse binary patterns. This sparsity is introduced by the binary weights and random weight assignment in the initialization stage of the HTM. We propose the alternative deterministic method for the HTM initialization stage, which connects the HTM weights to the input data and preserves natural sparsity of the input information. Further, we introduce the hardware implementation of the dete…
Emerging non-volatile memory (NVM) technologies offer unique advantages in energy efficiency, latency, and features such as computing-in-memory. Consequently, emerging NVM technologies are considered an ideal substrate for computation and storage in future-generation neuromorphic platforms. These technologies need to be evaluated for fundamental reliability and security issues. In this paper, we present \emph{NeuroHammer}, a security threat in ReRAM crossbars caused by thermal crosstalk between memory cells. We demonstrate that bit-flips can be deliberately induced in ReRAM devices in a crossb…
In a data-driven economy, virtually all industries benefit from advances in information technology -- powerful computing systems are critically important for rapid technological progress. However, this progress might be at risk of slowing down if we do not address the discrepancy between our current computing power demands and what the existing technologies can offer. Key limitations to improving energy efficiency are the excessive growth of data transfer costs associated with the von Neumann architecture and the fundamental limits of complementary metal-oxide-semiconductor (CMOS) technologies…
Measurement of the Crystallization and Phase Transition of Niobium Dioxide Thin-Films for Neuromorphic Computing Applications Using a Tube Furnace Optical Transmission System
Logic-in-memory (LIM) describes the execution of logic gates within memristive crossbar structures, promising to improve performance and energy efficiency. Utilizing only binary values, LIM particularly excels in accelerating binary neural networks, shifting it in the focus of edge applications. Considering its potential, the impact of faults on BNNs accelerated with LIM still lacks investigation. In this paper, we propose faulty logic-in-memory (FLIM), a fault injection platform capable of executing full-fledged BNNs on LIM while injecting in-field faults. The results show that FLIM runs a si…
Reading several ReRAMs simultaneously in a neuromorphic circuit increases power consumption and limits scalability. Applying small inference read pulses is a vain attempt when offset voltages of the read-out circuit are decisively more. This paper presents an experimental validation of a three-stage calibration scheme to calibrate the DC offset voltage across the rows of the memristive crossbar. The proposed method is based on biasing the body terminal of one of the differential pair MOSFETs of the buffer through a series of cascaded resistor banks arranged in three stages: coarse, fine and fi…
Neural processing on devices and circuits is fast becoming a popular approach to emulate biological neural networks. Elaborate CMOS and memristive technologies have been employed to achieve this, including chalcogenide-based in-memory computing concepts. Here we show that nano-scaled films of chalcogenide semiconductors can serve as building-blocks for novel types of neural computations where their tunable electronic and optical properties are jointly exploited. We demonstrate that ultrathin photoactive cavities of Ge-doped Selenide can emulate the computationally powerful non-linear operation…
